Category Archive Technology

How to: emulate SQS Amazon queue for testing purpose

This article first appear on Medium here

Amazon Simple Queue Service (SQS) is a fully managed message queuing service from Amazon.

If you want to integrate it, you may also want to test if it works. SQS has probably a sandbox to test it but if you don’t want to get stuck searching for it, or you don’t want to get crazy with login procedures, you can use a mock service that acts as an SQS queue* .

Why do we need this? Why can’t we use a simple Wiremock that always responds “200, OK”?

You can’t do it because the response of SQS is a dynamic response dependent from the request, and it’s based on some MD5 calculation.

 
MD5 calculation

What we have done is an extension of Wiremock that always responds something like “200, OK”, with all the fields correctly calculated for SQS. It was both easy and not-so-easy, because every easy task it’s never a “It’s just …“.

In this case, it was more of a “it’s just not enough to…” — just spinning a docker container up or adding a class to the classpath would simply not do.

How To, Step by step guide

First of all we need to define a class that extends theResponseDefinitionTransformer class with two easy to implement methods:

applyGobally that returns true, means that “all the requests will receive this response”.

getName it’s a method that Wiremock needs. Then you must implement the real method that responds correctly, the method is transform

calculateResponse does the dirty job, and it’s a dirty method (I know it could be refactored…)

This method gets the body of the request, splits it because it’s an url-like body, and then calculates the MD5 of all the attributes, types and values according to SQS documentation.

updateLengthAndBytes it’s a method to update the MD5 calculation.

To be honest, we also created an additional class SimplifiedMessageAttribute:

After this step, you can build an XML document (the implementation of this is not in this post because it‘s cumbersome) and include the calculated fields (md5OfMessageBody and md5OfMessageAttributes). For other fields, requestId and messageId, we used hard coded values. The final result should be something like this:

 
Response Example for SQS

After this, you can build your project and obtain a jar.

How can we integrate this into Wiremock?

That’s not so easy. What I’ve found is that you have to move your jar in the same path of the Wiremock jar and start the Wiremock with:

Wiremock has no plugin mechanisms, so what you are doing with this command is:

with -cp “*” you are telling Java “take all the jars in this folder”

… your main class is here: com.github.tomakehurst.wiremock.standalone.WireMockServerRunner

… and use this extension: your.package.name.SQSMock

From this point onwards, you can use it directly, build a docker container or whatever you want.

IT is not an easy task and, from my point of view, every layer added to simplify things is a new topic to learn and understand.

During this journey we have learnt about SQS, wiremock, mvn, unit test and obviously Java. I want to thank:

  • Giuseppe Gaeta (Tech Lead)
  • Matteo Moci (Software Dev Engineer II)
  • Camillo Quatrini (QA Evangelist)
  • Tiziana Battiston (Travel Agencies Manager, proofreader)

Ref:

* There are some restrictions: such service doesn’t work for not-string attributes.

Record from Fedora via ffpmeg

I’m playing around a web radio (for your info http://www.radiocittaperta.it, I’m also a speaker on Sunday) and what I’m trying to achieve is something that:

  1. Create podcast for every shows, given the start and stop hour
  2. Enable the speaker to manually start and stop a recording of the show
  3. Enable the speaker to choose if overwrite the automatic record or not
  4. Upload to the radio website
  5. Create the page with the podcast

Those are quite complicated task to be done for a single IT resource, but Rome wasn’t built in a day. This post is about recording the audio streaming from the input line of the attached sound card, using a webpage to start-stop the record.

First of all the radio has two computer with Fedora, they are configured as twins, just for redundancy. In the apache installation I have also installed a wordpress base installation, this gives me the functionalities of a CMS (user access, lost password, plugin management) in order to focus only on the requirments. I have created a wordpress plugin with a simple frontend button:

 

Start –> Stop

 

The behaviour follow this path https://trac.ffmpeg.org/wiki/PHP :

  • php start a record using ffmpeg in background
  • the pid of the process is saved in the database
  • on client side (javascript in the browser) an ajax request check if the process is still alive
  • stop button kills process using pid number (“kill -9 <pid#>”)

There are several pain points to take into account when using this approach:

I started with arecord command, and pipe with lame library to save mp3 files, but seems that arecord could run only once per-resource (resource=sound card) and it also has a problem that I don’t know why: it’s stops after 1h 33m (ref. http://stackoverflow.com/questions/42457839/recording-with-arecord-stops-after-1h-33m-under-fedora-23) , so I switched to ffmpeg.

Apache user could not been able to use audio. This is a limitation on most of the distro, I had to add apache user to the groups:

audio,pulse,pulse-access and I’ve added also video groups (don’t know if it’s necessary)

command is:

usermod -a -G  video,audio,pulse,pulse-access  apache

Another pain point on Fedora was SELinux. SELinux is an utility enabled by default, to manage security policy. There are several ways to work around this: investigate and set the perfect policy for folders (a lot of them are shared-memory folders), set the SELinux to be more permissive, disable SELinux. I don’t want to tell you the perfect solution, I’ve disabled it, but I don’t want you to do so, it’s a security related situation and a recovery after a security disaster is something that you really don’t want.

The ffmpeg wrapper class I wrote is 

<?php

include_once ‘Executer.php’;

/**
* Description of FfmpegWrapper
*
* @author mbarsocchi
*/
class FfmpegWrapper {

public function startRecord($mountpoint = null, $name, $timeToRun = null) {
$timeStringToRun = “”;
$filename = $name . “.mp3”;
if ($timeToRun != null) {
$timeStringToRun = “-t ” . $timeToRun;
}
$command = “/usr/bin/ffmpeg -y -f pulse -i alsa_input.usb-Focusrite_Scarlett_Solo_USB-00.analog-stereo -loglevel fatal ” . $timeStringToRun . ” -c:a libmp3lame -b:a 192k ” . dirname(__FILE__) . DIRECTORY_SEPARATOR . “ripped” . DIRECTORY_SEPARATOR . $filename . ” </dev/null”;
$exec = new Executer();
$exec->execute($command, true);
$res[‘retcode’] = $exec->getRetCode();
$res[‘output’] = $exec->getOutput();
$res[‘pid’] = $exec->getPid();
return $res;
}

public function stopRecord($pid) {
$exec = new Executer();
$exec->stopPid($pid);
$res[‘retcode’] = $exec->getRetCode();
$res[‘output’] = $exec->getOutput();
return $res;
}

}

Executer is a support class that wraps the shell_exec command. This class is also capable of getting background=true or background=false flag. When background=true is setted, the pid of the process is saved into DB with some additional information, in my case the additional infos are the name of the mp3 recorded.

To stop the recording, I’ve used a brutally Kill -9 <pid>. I know I can redirect std-in in a file and use it as a file-socket to send the [q] command to ffmpeg (ffmpeg is an interactive tool, so you can send command while working, q is for “quit”). You can also choose a more elegant solution, I use this because I don’t want to mess around with left files… and stuff like this. 

On the page ajax call an endpoint to retrieve the runnig process, if there is a live process, the page has the button STOP with a async call to an endpoint that calls:

FfmpegWrapper:: stopRecord(<pid>)

If not, the button is START.

This solution works very well for my needs, and hope this post helps you in similar solution.