Docker and Docker Sync

Posted December 17, 2016

I have been a docker-machine user for some time now, and to be honest I loved it. Performance was as good as with Vagrant, I didn’t have many issues except the biggie that was my containers running out of space. Within the docker-machine VM they only have a finite amount of resource available so it required a manual touch to make them bigger.

Ever since the beta came out for Docker for Mac and Windows I have been dying to use it as my docker tool yet the performance issues just made it un workable.

Yet with the introduction of the new Magento 2 developer box I thought I would have a play with it. The new dev box images comes with unison built in and because of its nature the dev box has to work cross platform.

In my spare time I also play around with other projects that are not Magento specific so I wanted to experiment with using native docker but not hit the hurdle of performance issues.

This is then where I found docker-syncDocker Sync. What is amazing about docker-sync is that it can use either Unison or RSYNC. The benefit is that the files are then served directly from the container and the sync method makes sure that files between the container and the host are in sync.

Docker sync and its commands

Lets take a look at how this works. We start with installing docker-sync. Sorry if you are a windows user but its Mac and Linux only.

gem install docker-sync

One this has installed you should have new commands available via the terminal docker-sync this has some extra commands that come with it each of which I will describe below:

Commands:
  docker-sync clean           # Stop and clean up all sync endpoints
  docker-sync help [COMMAND]  # Describe available commands or one specific command
  docker-sync list            # List all sync-points of the project configuration path
  docker-sync start           # Start all sync configurations in this project
  docker-sync sync            # sync - do not start a watcher

Options:
  -c, [--config=CONFIG]        # Path of the docker_sync config
  -n, [--sync-name=SYNC_NAME]  # If given, only this sync configuration will be references/started/synced

We also get docker-sync-stack

Commands:
  docker-sync-stack clean           # compose down your app stack, stop and clean up all sync endpoints
  docker-sync-stack help [COMMAND]  # Describe available commands or one specific command
  docker-sync-stack start           # Start sync services, watcher and then your docker-compose defined stack

Options:
  -c, [--config=CONFIG]        # Path of the docker_sync config
  -n, [--sync-name=SYNC_NAME]  # If given, only this sync configuration will be references/started/synced

For now lets not worry about what each of these commands does, instead lets focus on setting up a docker environment that follows the principles of docker and is portable between environments.

Configuration files

It all starts with a docker-compose.yml file. This is the configuration file that we can consider production ready.

version: "2"
services:
  app:
    build: ../../Docker-Images/docker-apache-php-7
    container_name: 'app-container'
    ports:
      - "1740:80"
    depends_on:
      - database
      - redis

  database:
    image: mysql:5.6
    ports:
      - "3306:3306"
    environment:
      MYSQL_ROOT_PASSWORD: password123
      MYSQL_DATABASE: magento2
      MYSQL_USER: dbuser
      MYSQL_PASSWORD: password123

  redis:
    container_name: magento2devbox_redis_1
    restart: always
    image: redis:3.0.7

  rabbit:
    container_name: magento2devbox_rabbit_1
    restart: always
    image: rabbitmq:3-management
    ports:
      - "8282:15672"
      - "5672:5672"

  elasticsearch:
    container_name: magento2devbox_elastic_1
    restart: always
    image: elasticsearch:1.7
    ports:
      - "9200:9200"

At first glance it looks like we have a lot going on here. Actually all we are doing is setting up 4 containers that will handle the application ( PHP + Apache ), the database ( MySQL ), Redis for cache and Elasticsearch for well search.

So far its pretty stock docker configuration and in theory this could be used in production. Now lets change it up and add another docker configuration file docker-compose-dev.yml

version: "2"
services:
  app:
    volumes:
      - simplest-sync:/var/www/html:rw # will be mounted on /var/www

# that the important thing
volumes:
  simplest-sync:
    external: true

All we are doing here is defining a volume that we want our app container to mount. Obviously in production we would not use a volume like this as we would compile the data or use another method. Yet for now you can see that in app -> Volumes we define the name and the path to mount and then finally we create the volume.

Next up we need to tell docker-sync some information and we does this by create a docker-sync.yml file:

syncs:
  simplest-sync: #tip: add -sync and you keep consistent names als a convention
    src: './app'
    dest: '/var/www/html'
    sync_host_port: 10872
    sync_strategy: 'unison'
    sync_user: 'www-data'
    sync_userid: '33'
    # sync_group: 'www-data'
    # sync_groupid: '33'
    sync_args:
      - "-ignore='Path .idea'"
      - "-ignore='Path .git'"
      - "-ignore='BelowPath .git'"
      - "-ignore='Path var/cache/*'"
      - "-ignore='Path var/sessions/*'"
      - "-ignore='Path node_modules/*'"

To recap we now have 3 files that are part of our project. We have docker-compose.yml a docker-compose-dev.yml and a docker-sync.yml. What is really good with this setup is that the volumes get added only via docker-compose-dev.yml and the sync file. What this means is that we have the ability to keep the same main composer file and push it up through our pipeline without having to worry about how files get added to the container.

Running docker-sync

Now when I first read the documents I read that I should be using docker-sync-stack start as the entry point to running my container. All this command does is wrap docker up commands and the sync start command. Yet in reality what I found is that there are at times issues with the sync container falling out of sync or crashing and stopping the stack and re starting the stack started to take time and frustration out of my life.

What I found as a better way of working was to use 2 separate tabs and run:

docker-compose -f docker-compose.yml -f docker-compose-dev.yml up If you want its easy to add the -d option to demonize docker and run it within the background but at times I like to see the logs to make sure everything is working fine. By using -f alias of --file all we are doing here is starting up docker compose and passing in both our production and developer configuration files.

Next up we need to run the sync command. docker-sync start. What this is doing is running the docker sync process that will start the unison container and make sure that the mounting is done properly to pull over assets and put them into the container and handle the configuration and watching via unison of files to be synced.

So just to summarize what I am running, In two tabs I have docker-compose -f docker-compose.yml -f docker-compose-dev.yml up and docker-sync start.

Known problems

I won’t say that this process has been without issues. As I said before I hit issues running the full stack command as at times the sync container just failed and I had to spend time debugging to realize that I had stale files in the container. Yet I have found that running them separately makes it easier to spot when this happens.

The biggest cause of the sync container falling in my experience is running large filesystem changes. Think composer install here. I looked on the github issues page for this type of error and it is a fault of unison that it can’t sync fast enough causing unison to crash.

My repo

I create a git repo that contains my docker images that Ive been using for Magento 2 development docker images. Some are already in docker hub some are still just experimental that I am playing with. There is also the beta image from the Magento 2 team that can be found somewhere on the internet. Don’t get me wrong its a good image and helped me over come issues with unison and permissions but docker is made for containerization and not monolithic vagrant machines and thats why I went down the route of creating containers per logical service.

You may also find these related posts interesting: All hail Xdebug and lets let var dump die A new look and a cleanup of content plus good bye github pages. Developer Book Club Patching Magento 2 vendor directory