Developing Sugar on Docker

TLDR

Developing Sugar on Docker, get up to speed now!

The long version

What are Docker containers?

If containers for you are more than the two pictures below… then you are in the right place!

As you might be aware, there has been and there continues to be a lot of hype around containers. The hype really started with Docker, but containers have been around for a while (more or less), in a different shape or form. If you are not yet familiar with containers or are confused about the difference between a virtual machine and a container, this blog post from Docker might be useful to you.

Personally, I was deploying container-based virtualised single purpose hosting environments (now labelled micro-services?) for my customers, back when it was not so trendy. It was about 10 years ago when I was leveraging OpenVZ. OpenVZ is slightly different (eg: it has state/persistent storage for every container), but in the end you can run multiple lightweight containers that share the custom linux kernel of their host.

What Docker has managed to do differently is make infrastructure deployments repeatable and portable. An application can be shipped together with its Docker container directives, providing the same exact configuration across different host environments. The infrastructure looks more like code, which simplifies the process of automating and delivering software deployments. Teams using Docker found that the lines between development and operations blurred even further, and guess what? DevOps was born!

Docker has made it easy to get up and running with a technology for development purposes. A developer just needs Docker installed on his or her local machine (or on a linux VM) and a “Dockerfile”, to get an application up and running. If he or she needs to replace a single component of the infrastructure for testing purposes, they simply swap a few directives and re-build. The same application is executing on top of a slightly different platform. How awesome and easy is that?

With the always increasing pace of software development innovation required today, and with applications becoming more and more decoupled from each other (and at the same time integrated), the deployment velocity needs to keep up as well to be able to quickly deliver innovation across the board on a multitude of heterogeneous backends.

This need for a fast deployment velocity can be fulfilled with the aid of flexible containers and orchestration technologies when their delivery is seamlessly baked into the traditional software development pipeline. Obviously containers are not the silver bullet or the answer to every known question, but they are currently applicable to a vast set of different workloads and use cases.

Docker in production

At this stage, I do not have direct experience with Docker in production environments, but, as with any technology, there is always someone willing to share their horror stories…

Jokes aside; with the increase in popularity of orchestration, the extensive backing and support from various cloud providers and how lightweight it is, I am convinced Docker is the way to go for micro-services deployment and orchestration in production environments as well. As a matter of fact, container-technology adoption is constantly growing across the board and not just for micro-services.

Sugar on Docker

Enough said about the containers topic. Here it is, my attempt at creating a set of development stacks for Sugar with Docker.

I have to thank one of our long-standing partners Cédric Mourizard from Synolia for the initial work (read more in his previous blog post). The Sugar Docker setup has been polished here and there over the past 6 months, and it is now to a point where I only run Sugar locally with this solution.

I highly encourage you to give it a try. Have a look at how Docker can fit in your development process, and, if it does not yet fit into it, think about how it could help you streamline your overall software delivery pace and capabilities.

Flavours used

Let’s give a quick overview of the various components used. When using *nix I’m a Debian kind of guy. So that’s that.

LAMP stack, Elasticsearch and Redis on top of Debian.

Stack components

It might evolve over time, but for now the main containers are:

• Apache load balancer – Load balances requests between the cluster of Apache PHP web servers, round robin
• Apache PHP web server – Web server(s)
• MySQL database – Database
• Elasticsearch – Sugar search engine
• Redis – Two purposes: Sugar object caching service and PHP Session storage/sharing service
• Cron – Sugar background scheduler processing. Note that this is enabled immediately and it will run cron.php as soon as the file is available, and it will attempt to do so every 60 seconds since its last run
• Permission – Sets Sugar instance permissions correctly and then terminates

The application files are shared on the “cluster” (if you can call it that) through a directory based mounted volume to offer persistent storage. We also have various persistent storage directories within ./data/ for all the other software components that need them.

Based on the selected stack (within the ./stack/ directory) you will be able to pick and choose the various versions of the stack components available, and to decide if you want a load balancer and multiple web servers or just a single web server during initial development phases. Because the Docker containers are stateless, it is possible to run the same Sugar installation (stored on the persistent volume) either with PHP 7.1 or PHP 5.6 and swap between versions within seconds.

Current software versions

Currently, there are a few stack combinations available that run the supported platform for Sugar 7.9.

Containers are available with PHP 7.1/5.6, MySQL 5.7, Elasticsearch 1.7.5 and the latest Redis. When selecting a stack, it is possible to run it as a single web server or as two web servers behind a load balancer.

Sugar 7.10 is not available for on-site hosting but it can be downloaded for local development, so, I have prepared a single stack leveraging PHP 7.1, compatible with the required matching platform. The main difference in stack requirements between Sugar 7.10 and earlier versions is the Elasticsearch version 5.4 requirement. Elasticsearch 5.4 seems to be more memory hungry, so be prepared for that.

Finally there is an additional stack for 7.9 upgrades, running both Elasticsearch 1.7.5 and 5.4, to allow future testing of upgrades to the Sugar Spring ’18 release.

Getting up and running

Most of the details you require to get up and running are within the README.md of the git repository. In terms of where to run this setup, you have the option of leveraging docker-machine or to run your own VM, as currently the native Docker for Mac does not perform well with the mounted volumes for persistent storage.

I personally use a Debian VirtualBox VM on a Macbook. If you want to run the same VM as I do, I have prepared a VirtualBox appliance export. You can find the appliance on this repository, as well as more details on how to deploy it on your Mac. It runs a NFS share as well, so that you can mount the directory locally and code with your preferred editor.

During my process, I’ve tried also Vagrant to help with automatic deployment of the VM instead of building an appliance, but I did not obtain the same performance footprint (not even close). So I abandoned that route long ago.

Now you are up to speed with Sugar on Docker, and you might have realised how containers can help your business gain velocity.

Feel free to let us know your thoughts, by leaving some comments below!

Adding a floating frame into Sugar

Sugar’s Single Page Architecture

Sugar relies on a single page architecture (SPA) for the Sidecar framework. But when a user navigates across the Sugar application (for example, when switching to another Sugar module), while the page is not refreshed, you will find that the majority of the HTML on the page is still re-rendered based upon the newly selected layout. This is done to not only change the style or configuration of the view (ex. List View → Record View) but also to update the context and configuration for the dashboard panel.

But not everything changes – the footer and header of the application remain fixed so they can serve as navigational anchors for Sugar users. This is an important use case but there are certainly others.

Telephony integration scenarios

A common set of integrations that are built for Sugar involve integrating a phone system for use by customer service organizations. This can vary from simple “click to dial” softphone functionality to full blown multi-channel call center capability where an agent handles phone, SMS, and e-mail inquiries at the same time.

A typical in-bound call process follows:

Using the Sugar REST PHP Client

Sugar REST PHP Client

A new open source library for working with Sugar 7’s powerful REST API has just been published! You can view the library in our GitHub account here: https://github.com/sugarcrm/rest-php-client

Full instructions for installation, usage, current included API Endpoints, and ways to contribute can all be found in the GitHub Wiki on the repository.

Who should use it?

The Sugar REST PHP Client was built initially to make working with Sugar instances easier for some of our internal tools. I wanted to provide a quick, easy, and object oriented, way to access Sugar 7’s REST API that could be used by all PHP developers. Any developer who is developing PHP applications that integrate with Sugar 7 should be interested in using the new library. The simplified architecture takes away the hassle of setting up and managing Curl connections, and allows the developer to focus on what matters most, which is working with the data in their Sugar application.

Using SugarCRM XHProf Viewer to profile Sugar server-side performance

Sugar Performance Engineer Vadzim Ramanenka shares some tips for profiling Sugar code using our newly launched SugarCRM XHProf Viewer open source project.

Profiling Sugar softly

Whenever you encounter that something is not working as fast as you would like, you need a way to look “under the hood” and understand what exactly is slowing things down. Sugar 7.7.0 added a built-in capability to collect detailed performance profiling data and store it in files. Recently, we released an open source SugarCRM XHProf Viewer project which can be used to visualize performance bottlenecks.

Example of a XHProf call graph diagram

Read on to learn how to configure both Sugar and the Viewer in a few simple steps.

Announcing XHProf Viewer, REST PHP Client, and Sugar Repairs Module open source projects

At SugarCRM, we have been accelerating the rate at which we share technology with the Sugar Developer community. Back in June at UnCon, we shared more open source code examples and tools than ever before. In April, we announced Sucrose Charts and the Sugar REST Harness. I am pleased to announce that we have open sourced three more projects under the Apache-2 license!

UnCon 2016 slides and code are available!

As promised, slides and code from UnCon are now available.  Video editing is still being worked on but videos of UnCon general sessions will be posted as soon as they are available. My Suga colleagues really outdid themselves this year!

UnCon 2016 Slides

All the slides from each of the general and breakout sessions have been posted in the UnCon community.

This is a great opportunity to refresh your memory or review any of the presentations that you happened to have missed this year. There are 28 presentations to go through.

Use these presentations as an aide to help you present what you learned at UnCon to your own colleagues!

UnCon 2016 Code

All example code shown at UnCon is available in the UnCon Github repository in the 2016 branch.  All the example code in this repository is licensed under Apache 2.0 unless otherwise noted.

There are a ton of code examples to learn and try out for yourselves. We presented and shared at least 2X more code than last year!

SLOC for UnCon 2016 Branch (as of June 28th)

Language	files	blank	comment	code
PHP	69	455	747	2,806
JavaScript	25	283	377	1,709
CSS	1	294	25	1,019
XML	7	0	0	347
Handlebars	16	5	63	204
JSON	3	4	0	73
LESS	1	0	3	2
SUM:	122	1,041	1,215	6,160

We also showed open source projects that exist in other Github repositories such as Sucrose.io and Sidecar Debugger Tool.

New releases of Sugar Test Tools for Sugar 7.7

To coincide with the release of Sugar 7.7, the Engineering team has released updated versions of Unit Tests and Performance tools.

Requesting access to Sugar Test Tools

Sugar Test Tools are in private Github repositories within the SugarCRM Github organization. Requesting to have your Github account added to the SugarCRM Github organization is easy, just fill out this form.  Visit the Developer Tools section of this site to learn more.

How to perform Data Anonymization for Sugar development

Here is another guest post from Emmanuel Dyan from the Elite SugarCRM Partner iNET Process. In it he addresses a common Sugar project requirement using an open source tool developed by iNET Process.

The problem that we will try to solve with this post is:

How do we make sure that we are never developing using actual customer data but, at the same time, work with data that reflects reality?

Data Anonymization

Usually, when we work on a customization project for a customer you have a minimum of 3 different environments: Development,  User Acceptance Testing (UAT), and Production. To make sure that we work in optimal and consistent conditions, we usually copy the database from one environment to another (preferably from production to other environments). Doing this type of manipulation has multiple drawbacks, including:

• We have to collect a Database Dump which means that it contains unencrypted raw data. What would happen if we mistakenly expose this dump to someone who is unauthorized?
• We have to test some functionality to make sure that it works. What would happen if we test a Campaign that sends thousand of e-mails … to the … actual customers of our customer?

Anonymizing the data is the best practice to avoid “playing” with customer data and to keep their trust in us.

The challenge with anonymizing data is figuring out how to overwrite the data with something that is completely unrecognizable. For example: “John Doe” will become “voluptatem accusantium“. His job title becomes “doloremque” and his country “magnam“. His phone number will become “569898520114457878744778” instead of “+123456789“.

New SugarCRM sponsored open source projects!

At SugarCRM Engineering, open source is part of our DNA.

We build Sugar on open source technology and open standards. We host and run our software on open source platforms in Sugar On-Demand. And all of our developers use open source tools for developing Sugar and implementing CRM projects for our customers.

As much as we benefit from open source software, we are obliged to give back. We host a number of open source projects in our SugarCRM Github organization that are actively maintained by SugarCRM employees. We encourage everyone in the Sugar Developer community to collaborate with us on these projects!

Read more to find out about some brand spanking new SugarCRM open source projects.

Using Composer with Sugar PHP development

Jun 14 2017 UPDATE

The latest versions of Sugar includes native support for Composer. You can customize the core ./composer.json to add and manage new 3rd party library dependencies. Customizing ./composer.json is only currently supported for Sugar On-Site. This method of using Composer is preferred over the one described by Emmanuel in this blog post due to the risk of conflicting dependencies through the use of two Composer projects within the same application.

Please view our Composer documentation in the Developer Guide for more details!

Here is a guest post from Emmanuel Dyan from iNET Process, an Elite SugarCRM Partner, and an active Sugar Developer community member as well as an open source advocate.

Are you interested in posting on the Sugar Developer Blog? Contact developers@sugarcrm.com with your idea.

Managing libraries and dependencies within Sugar

Sugar Partners sometimes need to share libraries between various Sugar development projects. It is also required that we use the custom directory structure for Sugar customization. Sometimes developers will just copy and paste libraries or other dependencies into the custom directory that they need to use for each of their Sugar projects.  But there are better ways to manage your code’s dependencies.

Composer is the best way to manage your software dependencies for PHP, so much so that it is de facto standard that PHP developers are adopting (including the team at SugarCRM). In this blog post, we will explore how you can use Composer to easily manage libraries used with Sugar development.

If you haven’t used Composer before, then check out Composer’s Getting Started guide for installation instructions.

Installing libraries using Composer

Let us suppose we have a project where we need to read YAML files. The best component to do with PHP is symfony/yaml. So, instead of downloading it and unzipping it into any directory, we can use Composer by running :

$ cd custom/
$ composer require symfony/yaml

This will generate a composer.json file that contains our new dependency and download the symfony/yaml library into the vendor directory.  However, we also need to make some additional edits to this file.

Here is an example of a complete config file that we can use with Sugar.

custom/composer.json

{ 
   "require":{ 
      "symfony/yaml":"^2.8" 
   }, 
   "require-dev":{ 
      "inetprocess/libsugarcrm":"^1-beta" 
   }, 
   "autoload":{ 
      "psr-0":{ 
         "":"include/lib/" 
      } 
   }, 
   "autoload-dev":{ 
      "psr-4":{ 
         "Inet\\SugarCRM\\Tests\\":"vendor/inetprocess/libsugarcrm/tests/" 
      } 
   }, 
   "config":{ 
      "platform":{ 
         "php":"5.4" 
      } 
   } 
}

Importantly, using the config.platform.php setting above allows us to ensure everything downloaded is compatible with PHP 5.4 as required by the Sugar version we are using.

Since we want to follow PHP best practices, we can also add a folder to the autoloader to load our own PSR compliant classes and methods.

For example, we may create code with the Inet\MyLib namespace that we store at custom/include/lib/Inet/MyLib.

Also in the above example, we have included a dev dependency for inetprocess/libsugarcrm. This is an open source library that is helpful for writing Sugar unit tests for our customizations.

Adding Composer’s Autoloader using Utils Extension

Finally, we’ll ask Sugar to use Composer’s autoloader by using Sugar’s Utils Extension to include an additional autoloader.

custom/Extension/application/Ext/Utils/composer.php

<?php
require_once(__DIR__ . '/../../../vendor/autoload.php');

With this in place we can share everything between modules!

Using sugarcli to quickly set up Composer

If you want to automatically setup Composer for your Sugar instance, just use … sugarcli!

Follow instructions on Github to download it, install it, and launch:

./sugarcli.phar code:setupcomposer --path <sugarcrm_path> --do
# Or, if it's in the path: sugarcli code:setupcomposer --path <sugarcrm_path> --do

That will check if the file exist, complete the installation or create both the files. It can even reinstall Composer if you want to start from scratch.