Loïc Faugeron Technical Blog

Towards CQRS, Search Engine 18/05/2016

TL;DR: A Search Engine component can help you get the CQRS "Query" part right.

The Command / Query Responsibility Segregation (CQRS) principle states that "write" and "read" logic should be separated. For example a single "Publisher" server (write) with many "Subscribers" servers (read) is a macro example of applying this principle, and an API that defines a read endpoint free from write side effects is a micro example of applying it.

Because it's hard to go from a regular mindset to a CQRS one, we've seen in the last article how we can use the Command Bus pattern to help us get the Command part right.

The "Query" equivalent of the Command Bus would be the Repository design pattern.

Note: Here's some nice articles about this pattern:

However repositories can grow into an object containing many methods like findByName, findAllInDescendingOrder, findByNameInDescendingOrder, etc. To avoid this, we can combine it with the Specification design pattern: our Repository would only have one single search method taking a Criteria object that describes our query parameters.

Note: For more articles on the topic, see:

In this article, we'll build a private "Search Engine" component to help us get the Query part right.

Requirements

All projects are different, and while the feature described here might have some similarity with other projects requirements, there's a big chance that creating a single common library might prove too hard.

So instead we'll create a "private" Component for our project: it's going to be decoupled, allowing it to become a library on its own just in the unlikely event it turns out to be the "Universal Search Engine Component".

Our requirements will be the following: we need to create an endpoint that allows us to search "profiles", with the following features:

Here's a HTTP Request example:

GET /v1/profiles?name=marvin&page=42&per_page=23&sort=-name HTTP/1.1
Accept: application/json

Note: sort's value can be either name (the default) or -name (changes the direction to be descending).

And here's a HTTP Response example:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "items": [
        {
            "name": "Arthur Dent"
        },
        {
            "name": "Ford Prefect"
        },
        {
            "name": "Trillian Astra"
        }
    ],
    "page": {
        "current_page": 1,
        "per_page": 10,
        "total_elements": 3,
        "total_pages": 1
    }
}

The Search Engine component

In order to satisfy the above requirements, we need to create a flexible Search Engine component that can accept any of those parameters. That can be done in one interface and a few (4) value objects.

First, we can have a SearchEngine interface:

<?php

namespace AppBundle\Search;

use Porpaginas\Result;

interface SearchEngine
{
    public function match(Criteria $criteria) : Result;
}

Note: We're using porpaginas, a library that makes paginated result a breeze to handle. Find out more about it here.

A Criteria is a value object, composed of:

It can be constructed using the query parameters:

<?php

namespace AppBundle\Search;

use AppBundle\Search\Criteria\Filtering;
use AppBundle\Search\Criteria\Ordering;
use AppBundle\Search\Criteria\Paginating;

class Criteria
{
    public $resourceName;
    public $filtering;
    public $ordering;
    public $paginating;

    public function __construct(
        string $resourceName,
        Filtering $filtering,
        Ordering $ordering,
        Paginating $paginating
    ) {
        $this->resourceName = $resourceName;
        $this->filtering = $filtering;
        $this->ordering = $ordering;
        $this->paginating = $paginating;
    }

    public static function fromQueryParameters(string $resourceName, array $queryParameters) : self
    {
        return new self(
            $resourceName,
            Filtering::fromQueryParameters($queryParameters),
            Ordering::fromQueryParameters($queryParameters),
            Paginating::fromQueryParameters($queryParameters)
        );
    }
}

The Paginating value object takes care of the page parameter (e.g. 1) and the per_page parameter (e.g. 10):

<?php

namespace AppBundle\Search\Criteria;

class Paginating
{
    const DEFAULT_CURRENT_PAGE = 1;
    const DEFAULT_ITEMS_PER_PAGE = 10;

    public $currentPage;
    public $itemsPerPage;
    public $offset;

    public function __construct(int $currentPage, int $itemsPerPage)
    {
        $this->currentPage = $currentPage;
        if ($this->currentPage <= 0) {
            $this->currentPage = self::DEFAULT_CURRENT_PAGE;
        }
        $this->itemsPerPage = $itemsPerPage;
        if ($this->itemsPerPage <= 0) {
            $this->itemsPerPage = self::DEFAULT_ITEMS_PER_PAGE;
        }
        $this->offset = $this->currentPage * $this->itemsPerPage - $this->itemsPerPage;
    }

    public static function fromQueryParameters(array $queryParameters) : self
    {
        $currentPage = $queryParameters['page'] ?? self::DEFAULT_CURRENT_PAGE;
        $maximumResults = $queryParameters['per_page'] ?? self::DEFAULT_ITEMS_PER_PAGE;

        return new self($currentPage, $maximumResults);
    }
}

The Ordering value object takes care of the sort parameter (e.g. -name):

<?php

namespace AppBundle\Search\Criteria;

class Ordering
{
    const DEFAULT_FIELD = 'name';
    const DEFAULT_DIRECTION = 'ASC';

    public $field;
    public $direction;

    public function __construct(string $field, string $direction)
    {
        $this->field = $field;
        $this->direction = $direction;
    }

    public static function fromQueryParameters(array $queryParameters) : self
    {
        $column = $queryParameters['sort'] ?? self::DEFAULT_FIELD;
        $direction = self::DEFAULT_DIRECTION;
        if ('-' === $column[0]) {
            $direction = 'DESC';
            $column = trim($column, '-');
        }

        return new self($column, $direction);
    }
}

The Filtering value object takes care of all the other parameters:

<?php

namespace AppBundle\Search\Criteria;

class Filtering
{
    public $fields;

    public function __construct(array $fields)
    {
        $this->fields = $fields;
    }

    public static function fromQueryParameters(array $queryParameters) : self
    {
        $fields = $queryParameters;
        unset($fields['page']);
        unset($fields['per_page']);
        unset($fields['sort']);

        return new self($fields);
    }
}

With this we have a generic Search Engine. The next step is to provide an implementation.

A Doctrine implementation

All implementations of SearchEngine need to be able to handle many types of parameters (pagination, filtering, etc).

To avoid our Doctrine implementation to become a big ball of mud, we're going to split the work into Builders, which construct the DQL query using the QueryBuilder:

<?php

namespace AppBundle\DoctrineSearch;

use AppBundle\Search\Criteria;
use AppBundle\Search\SearchEngine;
use Doctrine\DBAL\Connection;
use Doctrine\DBAL\Query\QueryBuilder;
use Porpaginas\Result;

class DoctrineSearchEngine implements SearchEngine
{
    private $connection;
    private $builders = [];

    public function __construct(Connection $connection)
    {
        $this->connection = $connection;
    }

    public function add(Builder $builder)
    {
        $this->builders[] = $builder;
    }

    public function match(Criteria $criteria) : Result
    {
        $queryBuilder = new QueryBuilder($this->connection);
        foreach ($this->builders as $builder) {
            if (true === $builder->supports($criteria)) {
                $builder->build($criteria, $queryBuilder);
            }
        }

        return new DoctrineResult($queryBuilder);
    }
}

Here's the Builder interface:

<?php

namespace AppBundle\DoctrineSearch;

use AppBundle\Search\Criteria;
use Doctrine\DBAL\Query\QueryBuilder;

interface Builder
{
    public function supports(Criteria $criteria) : bool;
    public function build(Criteria $criteria, QueryBuilder $queryBuilder);
}

We're not going to execute the query immediately, we're instead going to return the QueryBuilder wrapped in a Result implementation: this will allow us to choose between getting all the profiles or only getting a subset:

<?php

namespace AppBundle\DoctrineSearch;

use Doctrine\DBAL\Query\QueryBuilder;
use Porpaginas\Result;

class DoctrineResult implements Result
{
    private $queryBuilder;

    public function __construct(QueryBuilder $queryBuilder)
    {
        $this->queryBuilder = $queryBuilder;
    }

    public function take($offset, $limit)
    {
        $queryBuilder = clone $this->queryBuilder;
        $queryBuilder->setFirstResult($offset);
        $queryBuilder->setMaxResults($limit);
        $statement = $queryBuilder->execute();

        return new IteratorPage($statement->getIterator(), $offset, $limit, $this->count());
    }

    public function count()
    {
        $queryBuilder = clone $this->queryBuilder;
        $subSql = $queryBuilder->getSql();
        $sql = <<<SQL
SELECT count(*) AS count
FROM (
    $subSql
) as sub_query
SQL
        ;
        $result = $queryBuilder->getConnection()->fetchAssoc($sql, $queryBuilder->getParameters());

        return $result['count'] ?? 0;
    }

    public function getIterator()
    {
        $queryBuilder = clone $this->queryBuilder;
        $statement = $queryBuilder->execute();

        return $statement->getIterator();
    }
}

Finally if a subset is asked, we need to provide our implementation of Page:

<?php

namespace AppBundle\DoctrineSearch;

use Porpaginas\Page;

class IteratorPage implements Page
{
    private $iterator;
    private $offset;
    private $limit;
    private $totalCount;

    public function __construct(\Iterator $iterator, int $offset, int $limit, int $totalCount)
    {
        $this->iterator = $iterator;
        $this->offset = $offset;
        $this->limit = $limit;
        $this->totalCount = $totalCount;
    }

    public function getCurrentOffset()
    {
        return $this->offset;
    }

    public function getCurrentPage()
    {
        if (0 === $this->limit) {
            return 1;
        }

        return floor($this->offset / $this->limit) + 1;
    }

    public function getCurrentLimit()
    {
        return $this->limit;
    }

    public function count()
    {
        return count($this->iterator);
    }

    public function totalCount()
    {
        return $this->totalCount;
    }

    public function getIterator()
    {
        return $this->iterator;
    }
}

Building our Query

Now that we have a fully functional Search Engine, we need to create Builders specific for our need. The first one will be about selecting profiles:

<?php

namespace AppBundle\Profile\DoctrineSearch;

use AppBundle\DoctrineSearch\Builder;
use AppBundle\Search\Criteria;
use Doctrine\DBAL\Query\QueryBuilder;

class SelectProfileBuilder implements Builder
{
    public function supports(Criteria $criteria) : bool
    {
        return 'profile' === $criteria->resourceName;
    }

    public function build(Criteria $criteria, QueryBuilder $queryBuilder)
    {
        $queryBuilder->select('name');
        $queryBuilder->from('profile', 'p');
    }
}

We also need to be able to order our results:

<?php

namespace AppBundle\Profile\DoctrineSearch;

use AppBundle\DoctrineSearch\Builder;
use AppBundle\Search\Criteria;
use Doctrine\DBAL\Query\QueryBuilder;

class OrderingBuilder implements Builder
{
    public function supports(Criteria $criteria) : bool
    {
        return true;
    }

    public function build(Criteria $criteria, QueryBuilder $queryBuilder)
    {
        $queryBuilder->orderBy(
            $criteria->ordering->field,
            $criteria->ordering->direction
        );
    }
}

Finally if a name filter is provided we want to apply it:

<?php

namespace AppBundle\Profile\DoctrineSearch;

use AppBundle\DoctrineSearch\Builder;
use AppBundle\Search\Criteria;
use Doctrine\DBAL\Query\QueryBuilder;

class ProfileNameFilteringBuilder implements Builder
{
    public function supports(Criteria $criteria) : bool
    {
        return 'profile' === $criteria->resourceName && isset($criteria->filtering->fields['name']);
    }

    public function build(Criteria $criteria, QueryBuilder $queryBuilder)
    {
        $queryBuilder->where('p.name LIKE :name');
        $queryBuilder->setParameter('name', "%{$criteria->filtering->fields['name']}");
    }
}

Usage example

Let's create our controller:

<?php
// File: src/AppBundle/Controller/SearchProfilesController.php

namespace AppBundle\Controller;

use Sensio\Bundle\FrameworkExtraBundle\Configuration\Method;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\HttpFoundation\Request;

class SearchProfilesController extends Controller
{
    /**
     * @Route("/api/v1/profiles")
     * @Method({"GET"})
     */
    public function searchProfilesAction(Request $request)
    {
        $criteria = Criteria::fromQueryParameters(
            'profile',
            $request->query->all()
        );
        $page = $this->get('app.search_engine')->match($criteria)->take(
            $criteria->paginating->offset,
            $criteria->paginating->itemsPerPage
        );
        $totalElements = $page->totalCount();
        $totalPages = (int) ceil($totalElements / $criteria->paginating->itemsPerPage);

        return new JsonResponse(array(
            'items' => iterator_to_array($page->getIterator()),
            'page' => array(
                'current_page' => $criteria->paginating->currentPage,
                'per_page' => $criteria->paginating->itemsPerPage,
                'total_elements' => $totalElements,
                'total_pages' => $totalPages,
            ),
        ), 200);
    }
}

And that's it!

Conclusion

Just like using and overusing the "Command Bus" pattern can help us learn more on how to get the "Command" part of CQRS right, using and overusing the "Repository" design pattern in combination with the "Specification" one can help us get the "Query" part right.

Building a private Search Engine component for our project is one way to achieve this.