Our blog is moving(!) and other exciting community updates

We’re getting a head start on our New Year’s Resolutions!  We’ve given our developer community a fresh, new look, and we feel ten pounds lighter.  Let me give you a tour!

First, we’ve migrated our blog from developer.sugarcrm.com to the Developer Community at https://community.sugarcrm.com/community/developer/pages/dev-blog.  We’re really excited about this change as the community is now your one-stop-shop for everything related to developing on Sugar.  We’ll temporarily be putting new posts on both blogs, but please update your bookmarks now.

New tutorial! How to register custom platforms in Sugar instances

If you have a REST API integration with Sugar, you are likely using a custom platform as part of that integration.  Beginning in the Winter ‘18 release, Sugar is enforcing a rule that all custom platforms must be registered in each Sugar instance where a custom platform is used.  For more details on the rule itself, see Matt’s earlier blog post: Unknown platforms to be restricted in Winter ‘18 release.

The bad news?  This change that requires all custom platforms to be registered probably affects you.

The good news?  There is an easy fix, and I’ve written a tutorial that walks you step-by-step through how to implement the fix.

Check out the tutorial here!

Developing Sugar on Docker

TLDR

Developing Sugar on Docker, get up to speed now!

The long version

What are Docker containers?

If containers for you are more than the two pictures below… then you are in the right place!

As you might be aware, there has been and there continues to be a lot of hype around containers. The hype really started with Docker, but containers have been around for a while (more or less), in a different shape or form. If you are not yet familiar with containers or are confused about the difference between a virtual machine and a container, this blog post from Docker might be useful to you.

Personally, I was deploying container-based virtualised single purpose hosting environments (now labelled micro-services?) for my customers, back when it was not so trendy. It was about 10 years ago when I was leveraging OpenVZ. OpenVZ is slightly different (eg: it has state/persistent storage for every container), but in the end you can run multiple lightweight containers that share the custom linux kernel of their host.

What Docker has managed to do differently is make infrastructure deployments repeatable and portable. An application can be shipped together with its Docker container directives, providing the same exact configuration across different host environments. The infrastructure looks more like code, which simplifies the process of automating and delivering software deployments. Teams using Docker found that the lines between development and operations blurred even further, and guess what? DevOps was born!

Docker has made it easy to get up and running with a technology for development purposes. A developer just needs Docker installed on his or her local machine (or on a linux VM) and a “Dockerfile”, to get an application up and running. If he or she needs to replace a single component of the infrastructure for testing purposes, they simply swap a few directives and re-build. The same application is executing on top of a slightly different platform. How awesome and easy is that?

With the always increasing pace of software development innovation required today, and with applications becoming more and more decoupled from each other (and at the same time integrated), the deployment velocity needs to keep up as well to be able to quickly deliver innovation across the board on a multitude of heterogeneous backends.

This need for a fast deployment velocity can be fulfilled with the aid of flexible containers and orchestration technologies when their delivery is seamlessly baked into the traditional software development pipeline. Obviously containers are not the silver bullet or the answer to every known question, but they are currently applicable to a vast set of different workloads and use cases.

Docker in production

At this stage, I do not have direct experience with Docker in production environments, but, as with any technology, there is always someone willing to share their horror stories…

Jokes aside; with the increase in popularity of orchestration, the extensive backing and support from various cloud providers and how lightweight it is, I am convinced Docker is the way to go for micro-services deployment and orchestration in production environments as well. As a matter of fact, container-technology adoption is constantly growing across the board and not just for micro-services.

Sugar on Docker

Enough said about the containers topic. Here it is, my attempt at creating a set of development stacks for Sugar with Docker.

I have to thank one of our long-standing partners Cédric Mourizard from Synolia for the initial work (read more in his previous blog post). The Sugar Docker setup has been polished here and there over the past 6 months, and it is now to a point where I only run Sugar locally with this solution.

I highly encourage you to give it a try. Have a look at how Docker can fit in your development process, and, if it does not yet fit into it, think about how it could help you streamline your overall software delivery pace and capabilities.

Flavours used

Let’s give a quick overview of the various components used. When using *nix I’m a Debian kind of guy. So that’s that.

LAMP stack, Elasticsearch and Redis on top of Debian.

Stack components

It might evolve over time, but for now the main containers are:

• Apache load balancer – Load balances requests between the cluster of Apache PHP web servers, round robin
• Apache PHP web server – Web server(s)
• MySQL database – Database
• Elasticsearch – Sugar search engine
• Redis – Two purposes: Sugar object caching service and PHP Session storage/sharing service
• Cron – Sugar background scheduler processing. Note that this is enabled immediately and it will run cron.php as soon as the file is available, and it will attempt to do so every 60 seconds since its last run
• Permission – Sets Sugar instance permissions correctly and then terminates

The application files are shared on the “cluster” (if you can call it that) through a directory based mounted volume to offer persistent storage. We also have various persistent storage directories within ./data/ for all the other software components that need them.

Based on the selected stack (within the ./stack/ directory) you will be able to pick and choose the various versions of the stack components available, and to decide if you want a load balancer and multiple web servers or just a single web server during initial development phases. Because the Docker containers are stateless, it is possible to run the same Sugar installation (stored on the persistent volume) either with PHP 7.1 or PHP 5.6 and swap between versions within seconds.

Current software versions

Currently, there are a few stack combinations available that run the supported platform for Sugar 7.9.

Containers are available with PHP 7.1/5.6, MySQL 5.7, Elasticsearch 1.7.5 and the latest Redis. When selecting a stack, it is possible to run it as a single web server or as two web servers behind a load balancer.

Sugar 7.10 is not available for on-site hosting but it can be downloaded for local development, so, I have prepared a single stack leveraging PHP 7.1, compatible with the required matching platform. The main difference in stack requirements between Sugar 7.10 and earlier versions is the Elasticsearch version 5.4 requirement. Elasticsearch 5.4 seems to be more memory hungry, so be prepared for that.

Finally there is an additional stack for 7.9 upgrades, running both Elasticsearch 1.7.5 and 5.4, to allow future testing of upgrades to the Sugar Spring ’18 release.

Getting up and running

Most of the details you require to get up and running are within the README.md of the git repository. In terms of where to run this setup, you have the option of leveraging docker-machine or to run your own VM, as currently the native Docker for Mac does not perform well with the mounted volumes for persistent storage.

I personally use a Debian VirtualBox VM on a Macbook. If you want to run the same VM as I do, I have prepared a VirtualBox appliance export. You can find the appliance on this repository, as well as more details on how to deploy it on your Mac. It runs a NFS share as well, so that you can mount the directory locally and code with your preferred editor.

During my process, I’ve tried also Vagrant to help with automatic deployment of the VM instead of building an appliance, but I did not obtain the same performance footprint (not even close). So I abandoned that route long ago.

Now you are up to speed with Sugar on Docker, and you might have realised how containers can help your business gain velocity.

Feel free to let us know your thoughts, by leaving some comments below!

How Sugar REST API versioning works

Sugar v11 REST API

In order to support some cool new dashboard features, we made some enhancements to our Dashboard APIs in the Fall ’17 release. So we added a new REST API version, v11, in that release in order to ensure clients and integrations that were using the existing v10 Dashboard APIs would continue to work properly. For more details on the v11 API, we recommend you check out a recording of our recent Fall ’17 release developer webinar and the related migration guide.

Today we won’t focus on version 11 specifically but instead on how the Sugar platform will be handling REST API versioning in the future. So read on!

Passing API version using HTTP Header

For REST v10 and our older web services frameworks, the API version was specified exclusively via the URL.

Example 1)

GET <sugar>/rest/v10/me

In the Fall ’17 (7.10) release, we introduced an additional way to specify the API version: using an HTTP header. This change allows resource URLs to be treated as unique identifiers to resources regardless of the API version in use.

The REST API version can now be specified via HTTP Accept header. The REST API version should be specified in either the URL or in the HTTP Accept header but not in both locations at once. This new method of specifying the API version in the HTTP Accept header better adheres to RESTful architectural principles. By essentially encapsulating the API version as part of media type, a single resource at a given URL could offer multiple representations using different data formats and API versions.

Example 2)

GET <sugar>/rest/me
...
Accept: application/vnd.sugarcrm.core+json; version=42

We recommend that client start specifying the API version in the Accept header instead of as a component of the URL. Sugar clients will adopt use of the Accept header in the future.

Future use of semantic versioning in REST API

As you probably have noticed, the REST API version is a property of the Sugar REST API as a whole and not individual endpoints.

We plan to implement a semantic versioning policy with MAJOR.MINOR format for the REST API next year.

The version will be specified in the request Accept header as MAJOR.MINOR (ex. 11.1) whereas the version in the endpoint url will be specified as MAJOR_MINOR (ex. 11_1).

The Sugar REST API MAJOR version will be incremented for each Sugar release where compatibility is being broken on any endpoint. The Sugar REST API MINOR version will be incremented for each Sugar release where an API endpoint is changed or additional API endpoints are being added without breaking compatibility.

Not every new Sugar release will result in new REST API versions. Our intent is to only add new features or make breaking changes to our REST API only when it is not possible or prudent (ex. for performance reasons) to do so with the current API.

When there is a breaking change, older API versions should continue to work as-is. For example, if a parameter was removed in newer version, the old version should still support that parameter.

What are breaking and non-breaking changes?

We want to be clear about what we consider to be an API breaking change and what is not.

Breaking Changes

• Removing or renaming an endpoint, a request parameter, or a fixed response field
  • For example, removing the GET /rest/v10/<module> endpoint or renaming the _acl field to _access_control.
• Removing support for a HTTP request method on an endpoint
  • For example, changing from PUT to PATCH method for updating records.
• Change in type for a request parameter
  • For example, a change from an Integer to Decimal value.
• Change in response field format
  • For example, a change in encoding for a response field from plaintext to base64.
• Semantic changes to request parameters
  • For example, a change from treating the % symbol as a literal instead of as a wildcard or a change in default value.
• Semantic changes to an API endpoint
  • For example, an endpoint that was documented to return all records by default and now returns no records by default.

Non-breaking Changes

• Metadata related changes in behavior
  • For example, when Sugar metadata (Vardefs, Viewdefs, etc.) changes are introduced during a Sugar release or as a result of a customization then REST API responses can vary as a result.
• Adding a new endpoint
  • For example, adding a GET rest/Accounts/:id/map endpoint.
• Adding a parameter to a request
  • For example, adding a boolean parameter called shorten_text that decides if long text fields are shortened to 200 characters.
• Adding a field to a response
  • For example, adding a _self field with link to resource URI in GET responses.
• Adding support for another HTTP request method
  • For example, adding support for PATCH method to an endpoint.

If you ever encounter a situation where there is a breaking change on a particular API version between future Sugar releases, then please report those issues via the Sugar Support portal.

Deprecation process for old API versions

Old REST API versions will continue to be supported until at least one year after they have been deprecated. We want to exercise good judgement on how quickly we remove old API versions but please understand that SugarCRM cannot indefinitely support an unlimited number of REST API versions.

What should you do?

1. When building new Sugar clients or integrations or updating existing ones, adopt the latest available REST API version possible in order to take advantage of all the latest features.
2. Use the HTTP Accept header to specify REST API version when using Sugar Fall ’17 release or later.
3. Monitor the release notes and documentation on each new Sugar release for deprecation notices on old API versions. Keep our customers safe by planning ahead!

Thorn: Test now, REST later

RESTful APIs have always been a big part of the Sugar application, as well as the most appropriate touchpoint for integrating with other enterprise systems. Unfortunately, for a long time they weren’t tested especially appropriately. Traditionally, REST APIs at Sugar were tested with PHPUnit, but only on the classes themselves – they weren’t actually tested via direct use of HTTP. Worse, the tests were messy – they often assumed existing data was present, and when it wasn’t, it was created and not deleted afterwards.

Thorn was envisioned as a solution to this problem. It’s intended to test Sugar’s RESTful API directly, abstract away unhelpful boilerplate, leave your database clean, and let you test like a user.

Thorn was open-sourced earlier this year after a brief internal testing period and is now available for any Sugar developer to use from github.com/sugarcrm/thorn. It is open source and available under the Apache License, version 2.0.

This initial post will focus on a technical and philosophical overview of Thorn. For usage information, see the Thorn website.

A new Sugar UX coming in Winter ’18

Please visit this post at the Developer Blog’s new home if you want to leave comments.

https://community.sugarcrm.com/community/developer/blog/2017/11/29/a-new-sugar-ux-coming-in-winter-18

 

SugarCRM is on a mission to empower our users to delight their customers. To that end, we are pleased to introduce the first phase of visual restyling, and give some tips on how to work with the new UX.

Our process in developing the new UX

Sugar 7 introduced the Sidecar framework with massive user interface improvements (called SugarUX) and a modernized architecture. Since then, we have been working hard on the next iteration of SugarUX built on the Sidecar framework. Our guiding principles came from our customers and their feedback on Sidecar. A competitive analysis and feedback from customers revealed three concerns; cluttered complicated UX, inefficient unguided layout, and a drab outdated user interface.

The SugarCRM User Experience team (led by Brian Ng) launched a rigorous heuristic evaluation of the web and mobile Sugar products. Issues were identified as we evaluated key use cases and new solutions were documented. This thorough understanding of the product, customers, and the industry has helped us identify which issues to address first.

Dark gray text on light gray background led to poor legibility. Heavy blacks and gray linen added to the dated look and feel.

Sugar Mobile had a dark and drab visual design. Buttons for key actions were positioned differently on key screens.

Unknown platforms to be restricted in Winter ’18 release

Sugar uses platforms to support the needs of multiple Sugar clients.  The Sugar REST API uses the platform parameter to indicate which platform is being used.  If you’d like a refresher on what the platform parameter is and how to use it, check out this blog post.  

In Sugar 7.9, we added a new Platform extension that we advised developers to start using in the Sugar 7.9 Migration Guide.  The Platform extension allows you to indicate a particular custom platform should be allowed when the disable_unknown_platforms configuration setting is on.

Changes coming in Winter ’18 release

In the Winter ’18 release, we will be preventing REST API access to Sugar from unknown platform types.

Sugar has a configuration setting disable_unknown_platforms that controls whether or not unregistered platforms are allowed to be used when logging in using the REST API. The current default value for disable_unknown_platforms is false. In the Winter ’18 release, we will be changing the default to true, which is how it is already reflected in the documentation.

If your integration uses a custom platform, this custom platform will need to be registered in each Sugar instance or your integration will break!

New tutorial! Intro to the Sugar REST API

Are you ready to build an integration with Sugar but not sure where to start?  You’ve come to the right place!

When you want to access or interact with information stored in Sugar, the REST API is a great place to start.  In this tutorial, you’ll learn how to authenticate to the Sugar REST API.  Then you’ll learn how to perform create, read, update, and delete (aka CRUD) operations.

Watch the video tutorial below or view the text-based tutorial at bit.ly/tutorial_rest.

How to create custom Sidecar routes

Have you ever found yourself wishing you could create a custom Sidecar user interface within your Sugar instance? Maybe you want to allow users to visit a URL that displays a custom view.

It turns out that creating a linkable URL (or route) to within the Sugar client is fairly simple. In this post, I’ll walk you through how to implement a new route in your Sugar instance that displays an alert.

Developing for Sugar On-Demand Only Releases

Sugar Fall ’17 (7.10) is now available!

You’ve heard all about this, but I’ll recap. SugarCRM’s fall 2017 release is a Sugar On-Demand only release. This is the first release that follows our new Sugar release process. Our on-premise customers will get a roll up spring release that includes features from this fall and subsequent winter release. So all of our customers will enjoy all of the same advanced features–just not all at once.

Here are some of the great developer resources available for the Sugar Fall ’17 release (also known as Sugar 7.10).

• Sugar 7.10 Developer Release Notes
• Sugar 7.10 Unit Tests (Requires Dev Tools sign-up for access)
• Sugar 7.10 Developer Guide that includes an updated Migration Guide
• Sugar 7.10 Developer Webinar Recording

Last, but not least, we will be providing developer downloads of On-Demand releases such as 7.10 that can be used by Sugar developers for code development and test purposes.