Getting Started with AWS Step Functions Part I

AWS came out with Step Functions a few years ago, and up until recently, I have not had the opportunity to dive in and give them a try. Yes, I could build my own pipeline or state machine, but the idea behind Step Functions is that it does most of the heavy lifting for you. That, and it ties into other AWS services. As such, I decided to dive into getting started, and looked at the demo options and walkthroughs that were available. None of them met my needs, so I rolled my own.

The idea is to see how I can create a Step Function that will run multiple loops, and call a Lambda function multiple times. What I wanted to test was the following:

  • Pass Variables into the Step Function and see how they are handled
  • Call a Lambda function multiple times
  • Create a loop using the Step Function DSL
  • Test output from Lambda and make a decision based upon it
  • Figure out any gotchas and how to trigger Step Functions

Let’s dive in. Now, this is the final result. It took me a few iterations to actually get to this point. Smarter people than I might be able to get it done on one go, but not I.

Lambda Code

I came up with a simple Lambda function written in Python 3.6. All that I wanted to do was to perform a loop with Step Functions, and then get output the values. Simple. And as you can see, this code is pretty simple. It could be streamlined, but it was quick and easy to write.

def lambda_handler(event, context):
    print('value1 = ' + event['key1'])
    print('value2 = ' + event['key2'])
    print('value3 = ' + event['key3'])
    taskresult = event.get('taskresult', None)
    if taskresult is None:
        count = 0
        count = taskresult.get('count', None)
    if count is None:
        count = 0
        count = count +1
    if count < 5:
        output = {
            'count' : count,
            'value1' : 'ThereIsNoSpoon';
        output = { 'value1' : event['key1'],
                    'value2': event['key2'],
                    'count' : count }
    return output

Now we need to move onto the body of what we are working on, and that would be the Step Function. Step Functions have their own language or domain specific language (DSL) that is used to define the state machine. I wanted more than just a “Hello World” example. The idea was to loop through a step functions. Make sure that I could call it multiple times, and then either go to a success or failed state

AWS Step Function Code

  "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
  "StartAt": "LambdaFunction",
  "States": {
    "LambdaFunction": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:aws-serverless-repository-hello-w-helloworldpython-1JQ8TEEDUAHCE",
      "ResultPath": "$.taskresult",
      "Retry": [
          "ErrorEquals": ["CustomError"],
          "IntervalSeconds": 1,
          "MaxAttempts": 2,
          "BackoffRate": 2.0
          "ErrorEquals": ["States.TaskFailed"],
          "IntervalSeconds": 30,
          "MaxAttempts": 2,
          "BackoffRate": 2.0
          "ErrorEquals": ["States.ALL"],
          "IntervalSeconds": 5,
          "MaxAttempts": 5,
          "BackoffRate": 2.0
      "Next": "ChoiceState"
    "ChoiceState": {
      "Type": "Choice",
      "Choices": [
          "Variable": "$.taskresult.value1",
          "StringEquals": "value1",
          "Next": "SuccessState"
          "Variable": "$.taskresult.count",
          "NumericLessThan": 5,
          "Next": "LambdaFunction"
      "Default": "FailState"
    "SuccessState": {
      "Type": "Succeed"
    "FailState": {
      "Type": "Fail",
      "Cause": "Invalid response.",
      "Error": "ErrorA"

The way that this code works is as follows. Everything works around States. So, you have to move from State to State. This is a key concept when it comes to Step Functions. Now, there are multiple State types, but I am not going to go into that now. The key factor is that you will go through and loops if a proper return value is not returned. Looking at it now, it looks like a bunch of gobbledygook. I am going to have to come back and write up how this works later.

This is what the visual representation looks like when viewed in the AWS Step Function page. There is a defined ‘Start’ and ‘Stop’. The other stages match what was named in the previous section. The code works to present a model that you can follow.

The cool think about AWS Step Functions is that they guarantee a run. And in a situation where you need to ensure that the code is run, and you need a guarantee. This is mostly due to the cost that is associated with it. Running Lambda that Triggers on SQS would be cheaper, but not as easy to ensure.

Back on with our stuff now. We are looking at how we execute the AWS Step Function. Now we need to execute it. Right now, I am not going to go into the logic around passing variables around. Needless, you will need to understand that when writing your own, and I am going to have to revisit it.

Execution. A couple of items to note.

  • Each execution has to have a unique name.
    • Note, this will bite you when you are testing, and think about this when executing it via automation.
  • It takes in an action just like Lambda, via json
  • Making a small change in the inputs can cause madness
  "key1": "value1",
  "key2": "value2",
  "key3": "value3"

The output will also be in json, and you can see the results in the visual display.

  "key1": "value1",
  "key2": "value2",
  "key3": "value3",
  "taskresult": {
    "value1": "value1",
    "value2": "value2",
    "count": 5

This is what the output looks like.

I will go into more detail on the breakdown of the Step Function in the next post. There is a lot to be covered, and this just scratches at the surface.

Writing Tech Blogs are Hard Work

I once read an article that said more people need to write technical blogs. That the problem with much of the technology field was that people did not write in-depth articles on the stuff that they are doing. And, if more people were able to take the time and post a blog entry here and there we would all be better for it. As nice as that sounds, I have to say that writing technical posts are difficult, time consuming, and can quickly go out of date.

First off, writing in general is not easy, and that is before you get to adding the technical part on top of it. There are a number of brilliant developers, system engineers, and devops people that can create some of the most complex and unique solutions to problems, and yet cannot begin to write the first bit of documentation or narratives to describe what it is that they have done. It is not that they are dumb, they just have not honed the writing skill, or just might not be proficient at it. Writing is like coding, if you don’t use it, you can lose it. Also, it is a skill that can be developed and honed overtime. Myself, my writing skills are rusty after taking a long time off from it, and I am trying to get back into the flow.

There are some people that recommend to become talented at writing, you need to be able to dedicate an hour a day to it, or 5 hours a week. And that is just the writing part. That does not tie in working on the technical elements that are needed to provide content for the audience you are trying to reach. Maybe a few years back when I did not have family obligations this would have been possible, but now I have to sneak in time here and there. And, for the casual tech blogger, this is not going to be the case. Unless you are doing a lot of writing for work, there is little time to develop your writing skills. This leads many people to turn away from writing a technical article even though they might have some of the best ideas, if only they could get them into a usable format.

Now, the next big item on why it is so hard to write technical articles is that gathering together the technical information is not easy. Don’t get me wrong, that is not to say that there are not a number of topic that you can write on. That is far from the truth. There are probably thousands of areas and topics that can be written about. But, just like writing, it takes time to gather that information. Now, there are a few high profile bloggers that are able to dedicate their jobs to writing technical blogs. Many of them are evangelist or full time employees whose job it is to talk about certain technical areas. That is great for them, and to honest, I am a bit envious. For the average person devops engineer or develop it is not so easy.

There are some companies that will allow you to blog on the work you are doing, but for most people that is not the case. So, on top of your full time job, you then most go and use your own resources and your own time to work on getting together the data, the code, and whatever else, to get together the technical information just to begin writing the article. Once the work has been done to vet the project or the topic that you are looking at writing about, you have to circle back around and figure out how it is that you want to get the information together to put it in a story for others to read. This goes back to item number one, and I have already mentioned how difficult even that first step is. Now we are taking it to a higher level by saying you do not just have to write a story, but that you must structure it around the technical information. 

So, after getting the data together and know what you are going to write about, you have to structure it. There are screen shots that have to be made, code snippets that must be shared, links to technical information that must be included. As easy as all of that sounds, it is much harder than it sounds. Figuring out how to clip an image or not make it be 4 megs in size, or how to get the images aligned so that it all looks correct. Then there is the question of where do you put your code you are going to share. How do you get the numbers showing on code? How do you enable syntax highlighting on the code samples? All of that is not easy. All of that can be overwhelming when you just want to write an article to share some information with other techies.

And the last fact of the matter is that you never know if all your efforts are for naught. You could write a great article, but if people don’t find out about it, then what do you do? Do you keep plugging away and hope people will stumble upon the articles you have written? Nobody wants to do a bunch of work for nothing. So, in recap, writing technical articles is difficult. It can be hard, and it is not always rewarding. I thank all the people that stick with it, but I understand all of those who don’t.

Making Adafruit Pixel Goggles

Let me start with the motivation for this project. On one hand, I had never really created a solution with a micro controller, and on the other I wanted to create a cool Halloween costume. The rest of my family was getting dressed up in steampunk outfits, and for a change, I wanted to add something to the mix. But, to do this, I was going to have to learn how to program this micro controller, solder, and how to get the code on the NeoPixel. I know that some people would laugh and say that this would be an easy project, but when you have never worked on this before, it is a completely different case.

Enough back story, time to walk through the build process. First, there are a the items that you will need to build out the entire unit. You will need a soldering iron. That is if you want to make it permanent. Otherwise, you will need a pair of goggles, 2 Adafruit NeoPixels, and an Adafruit Trinket. You also need a portable battery and a hookup to provide the power to the NeoPixel rings. I can break down the list and links, but they may go bad after a while. There is also a kit

There are instructions on the Adafruit website, but to be honest I found them a bit difficult to follow. There was some difficulty in understanding the instructions and what they were implying. They did have a very nice diagram that was of more use than the rest of the instructions. And, given everything, that is all you really need as far as laying out the wires.

First action is to solder on the JST-PH 2-Pin SMT Right Angle Connector to the Trinket. The USB power to the Trinket is not enough to power the NeoPixel Rings. As a result, you really need to hook up the connector so that you can provide a 5v power supply to the Trinket. Don’t make the same mistake that I did and hook up a 9v battery. This will be to much power, and the rings will only work sporadically. There is a technical reason for this, but I don’t remember electrical theory to explain it. I know enough to know that it will not work right.

Once you have the power supply soldered onto the back of the board, then the fun really begins. The power supply connectors are on the back and this is where you have to connect the adapter. There is a way to test the power supply, and that is to load the blink code onto the Trinket. I will make sure that is in my code sample. At this point, you should have a trinket that has a power supply hooked up to it, and the ability to push code to the Trinket.

That is one thing that I should have touched on. It was a huge pain to get the code to push to the Trinket when I first started working with it. I ended up have to install a very specific version of the Arduino development plaftorm, version 1.8.5, in order to get the libraries to work. In retrospect, I would have seen if I could have found a Trinket M0 board. That board is supposed to support python, and has more options when it comes to the IDE. However, as of this writing, I had to use a specific version of the Arduino IDE, but I was able to get all of it working. Just remember, that sometimes compiling the software first before pushing the button to sync makes it easier to get it all done in time.

The next thing to do was to hook up all the wires and see if it would work. Seeing as this was my first time working on the Trinket and with the NeoPixels I was not sure if I was going to get the wiring done correctly. To ensure that I was doing it right, I decided to use wires with (sic) roach clips to validate the circuit that I was going to make. At first I was intimidated by the circuit, and thought I was going to mess something up. In retrospect, it was pretty simple. 

First, lets talk about getting power to the rings. On the one side of the Trinket you have 2 connections. One connection is for outgoing power and the other is ground. There may be a technical reason for the name ground, but the simple part of it is that you need to create a complete circuit. This means the power goes from the power output on the board and goes to the power input on the NeoPixel. Then, then ground needs to come back to the board to complete the circuit. To add another ring to the mix, you take the second power connection from the NeoPixel and connect it to one of the power points on the other NeoPixel ring. Then hook up one of the ground connections to the second ground connection on the first ring. Bam! Power to both.

That is great and all, but you need to get the data to the rings. For this, the rings have a ‘data in’ and a ‘data out’ port. Unlike the power connection, this does not have to complete an entire circuit. You just select the outbound pin you want to use from the Trinket and connect it to the data in slot on the first board, then you connect the data out connection from that board to the data in on the second board. That is all. I thought there was more to it. There is not.

I will be the first to admit that I did a horrible job at soldering. It was my first time, and I had the temperature set to low, and then I had the wrong power level, so when I thought it was wrong, it turned out to be right, and I redid the entire thing. Don’t be me. Make sure you check the power 6 times and test after every solder. Unless, you are like a friend of mine. He had a job doing soldering, and he is a godlike at it.

The last part is the code. Ah, this is much more fun, and I think I might come back and revisit this later. Needless to say, the samples are here. I am a big fan of github and bitbucket for storing and sharing projects. That is a song and dance for a different day.

Micro controller Hacking coming soon

I have been hacking on my costume, and as such, have not gotten around to posting the code and pictures of the project that I am currently working on. As soon as Halloween roles by, I should be able to get that up here. Hopefully, you will enjoy the costume and the work that I have done on getting the goggles up and running. 

Normally, I would have tried to add it in chunks, but I have been learning how to solder, learning how to code this thing, and have been trying to get the rest of my costume configured. 

That being said, I am working on it. Oh, and for work I have been learning all the new nuances to python 3.6. Almost like learning another language from what python used to be. So, hold out for another few days, and I will share my code and some pics of the work as I have learned how to solder, and how to write a bit of code for an adafruit neopixel.

Issues with Ubuntu’s Startup Disk Creator for non-debian ISOs

Let me start with the scenario that led me into issues with the Ubuntu Startup Disk Creator. I had been running Ubuntu GNOME, a flavor of Ubuntu that was focused on a mostly vanilla install of Gnome on top of Ubuntu. Well, Ubuntu was finally getting rid of the Unity desktop, so the spin off that I had been using was no longer going to be updated. Fair enough.

I had Ubuntu installed, and I wanted to give Fedora a try (it has been a while), so I just needed to create a bootable USB stick. Normally I would use dd, but Ubuntu has a tool, and I thought sure, let’s try this gui tool. Quick and easy. That led me down the rabbit hole that you see here.

Ubuntu has a page dedicated to the topic. Create a bootable USB stick on Ubuntu. This page walks through using the Startup Disk Creator. It does mention using an Ubuntu ISO image, but what should work for one, should work for most any ISO. At least this is what I thought. It turns out that if you are not using a Debian based distro, then the application will fail silently. It just sits there and does nothing.

At this point, I could have just used dd and been done with it, but I wanted to find out what was going on with the application, and why it was not working. Let me add a quick note and say, I had not tried a Debian based ISO on the application. This was because I wanted to try out the latest Fedora, and had not bothered to pull down another ISO.

Finding the source code for the USB Creator took a bit longer than planned. The code is hosted on LaunchPad which uses Bazaar as its version control system. Having used multiple systems over the years, launchpad felt like a step back in time. Unless I missed something, there is no easy search within a project, the navigation is antiquated, and the look and feels leaves a bit to be desired.

My first thought was to create a bug on the issue. Even if I was going to fix it, I wanted to ensure that the issue was being tracked, and to see if anyone else had submitted a bug on the topic already. I was surprised when I went to the bug page, and found that the package had not been configured for bug reports yet. At this point, I was more frustrated than anything else, and had decided I was going to use dd to create the usb disk, but I wanted find out what the problem in the code was.

Digging through the code, I found that the core of the application is a few python scripts. Nothing wrong there. I am a huge fan of python for a number of reasons. So, I dug into the code and found the issue rather quickly to my surprise.


        if extension == '.iso':
            label = self._is_casper_cd(filename)
            if label:
                self.sources[filename] = {
                    'device' : filename,
                    'size' : os.path.getsize(filename),
                    'label' : label,
                    'type' : misc.SOURCE_ISO,
                if misc.callable(self.source_added_cb):
        elif extension == '.img':
            self.sources[filename] = {
                'device' : filename,
                'size' : os.path.getsize(filename),
                'label' : '',
                'type' : misc.SOURCE_IMG,
            if misc.callable(self.source_added_cb):

The issue is on line 42. The application checks to see if the iso file ‘is_casper_cd.’ This check returns `None` if it does not follow this format. A simple exception could have given the end user some sort of idea about what the issue was, but instead, it fails silently.


    # Device manipulation functions.
    def _is_casper_cd(self, filename):
        for search in ['/.disk/info', '/.disk/mini-info']:
            cmd = ['isoinfo', '-J', '-i', filename, '-x', search]
                output = misc.popen(cmd, stderr=None)
                if output:
                    return output
            except misc.USBCreatorProcessException:
                # TODO evand 2009-07-26: Error dialog.
                logging.error('Could not extract .disk/info.')
        return None

The fact that this code fails silently, the code base does not provide for bug features, and that there is no documentation on this bug/feature is problematic. A simple review of this should have caught the problem. Also, as you can see in the previous section, they do log an error when the data can not be extracted, but they do nothing if the file format does not exist.

At a minimum they could have thrown an error. Another option would have been to continue with the disk write, but to skip the label information. Otherwise, give the end user a clue.

In the end I just used good old fashioned `dd`.

user@host$ sudo dd bs=4M if=/path/to/archlinux.iso of=/dev/sdx status=progress oflag=sync