Getting Started with AWS Step Functions Part I

AWS came out with Step Functions a few years ago, and up until recently, I have not had the opportunity to dive in and give them a try. Yes, I could build my own pipeline or state machine, but the idea behind Step Functions is that it does most of the heavy lifting for you. That, and it ties into other AWS services. As such, I decided to dive into getting started, and looked at the demo options and walkthroughs that were available. None of them met my needs, so I rolled my own.

The idea is to see how I can create a Step Function that will run multiple loops, and call a Lambda function multiple times. What I wanted to test was the following:

  • Pass Variables into the Step Function and see how they are handled
  • Call a Lambda function multiple times
  • Create a loop using the Step Function DSL
  • Test output from Lambda and make a decision based upon it
  • Figure out any gotchas and how to trigger Step Functions

Let’s dive in. Now, this is the final result. It took me a few iterations to actually get to this point. Smarter people than I might be able to get it done on one go, but not I.

Lambda Code

I came up with a simple Lambda function written in Python 3.6. All that I wanted to do was to perform a loop with Step Functions, and then get output the values. Simple. And as you can see, this code is pretty simple. It could be streamlined, but it was quick and easy to write.

def lambda_handler(event, context):
    print('value1 = ' + event['key1'])
    print('value2 = ' + event['key2'])
    print('value3 = ' + event['key3'])
    taskresult = event.get('taskresult', None)
    if taskresult is None:
        count = 0
        count = taskresult.get('count', None)
    if count is None:
        count = 0
        count = count +1
    if count < 5:
        output = {
            'count' : count,
            'value1' : 'ThereIsNoSpoon';
        output = { 'value1' : event['key1'],
                    'value2': event['key2'],
                    'count' : count }
    return output

Now we need to move onto the body of what we are working on, and that would be the Step Function. Step Functions have their own language or domain specific language (DSL) that is used to define the state machine. I wanted more than just a “Hello World” example. The idea was to loop through a step functions. Make sure that I could call it multiple times, and then either go to a success or failed state

AWS Step Function Code

  "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
  "StartAt": "LambdaFunction",
  "States": {
    "LambdaFunction": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:aws-serverless-repository-hello-w-helloworldpython-1JQ8TEEDUAHCE",
      "ResultPath": "$.taskresult",
      "Retry": [
          "ErrorEquals": ["CustomError"],
          "IntervalSeconds": 1,
          "MaxAttempts": 2,
          "BackoffRate": 2.0
          "ErrorEquals": ["States.TaskFailed"],
          "IntervalSeconds": 30,
          "MaxAttempts": 2,
          "BackoffRate": 2.0
          "ErrorEquals": ["States.ALL"],
          "IntervalSeconds": 5,
          "MaxAttempts": 5,
          "BackoffRate": 2.0
      "Next": "ChoiceState"
    "ChoiceState": {
      "Type": "Choice",
      "Choices": [
          "Variable": "$.taskresult.value1",
          "StringEquals": "value1",
          "Next": "SuccessState"
          "Variable": "$.taskresult.count",
          "NumericLessThan": 5,
          "Next": "LambdaFunction"
      "Default": "FailState"
    "SuccessState": {
      "Type": "Succeed"
    "FailState": {
      "Type": "Fail",
      "Cause": "Invalid response.",
      "Error": "ErrorA"

The way that this code works is as follows. Everything works around States. So, you have to move from State to State. This is a key concept when it comes to Step Functions. Now, there are multiple State types, but I am not going to go into that now. The key factor is that you will go through and loops if a proper return value is not returned. Looking at it now, it looks like a bunch of gobbledygook. I am going to have to come back and write up how this works later.

This is what the visual representation looks like when viewed in the AWS Step Function page. There is a defined ‘Start’ and ‘Stop’. The other stages match what was named in the previous section. The code works to present a model that you can follow.

The cool think about AWS Step Functions is that they guarantee a run. And in a situation where you need to ensure that the code is run, and you need a guarantee. This is mostly due to the cost that is associated with it. Running Lambda that Triggers on SQS would be cheaper, but not as easy to ensure.

Back on with our stuff now. We are looking at how we execute the AWS Step Function. Now we need to execute it. Right now, I am not going to go into the logic around passing variables around. Needless, you will need to understand that when writing your own, and I am going to have to revisit it.

Execution. A couple of items to note.

  • Each execution has to have a unique name.
    • Note, this will bite you when you are testing, and think about this when executing it via automation.
  • It takes in an action just like Lambda, via json
  • Making a small change in the inputs can cause madness
  "key1": "value1",
  "key2": "value2",
  "key3": "value3"

The output will also be in json, and you can see the results in the visual display.

  "key1": "value1",
  "key2": "value2",
  "key3": "value3",
  "taskresult": {
    "value1": "value1",
    "value2": "value2",
    "count": 5

This is what the output looks like.

I will go into more detail on the breakdown of the Step Function in the next post. There is a lot to be covered, and this just scratches at the surface.

Writing Tech Blogs are Hard Work

I once read an article that said more people need to write technical blogs. That the problem with much of the technology field was that people did not write in-depth articles on the stuff that they are doing. And, if more people were able to take the time and post a blog entry here and there we would all be better for it. As nice as that sounds, I have to say that writing technical posts are difficult, time consuming, and can quickly go out of date.

First off, writing in general is not easy, and that is before you get to adding the technical part on top of it. There are a number of brilliant developers, system engineers, and devops people that can create some of the most complex and unique solutions to problems, and yet cannot begin to write the first bit of documentation or narratives to describe what it is that they have done. It is not that they are dumb, they just have not honed the writing skill, or just might not be proficient at it. Writing is like coding, if you don’t use it, you can lose it. Also, it is a skill that can be developed and honed overtime. Myself, my writing skills are rusty after taking a long time off from it, and I am trying to get back into the flow.

There are some people that recommend to become talented at writing, you need to be able to dedicate an hour a day to it, or 5 hours a week. And that is just the writing part. That does not tie in working on the technical elements that are needed to provide content for the audience you are trying to reach. Maybe a few years back when I did not have family obligations this would have been possible, but now I have to sneak in time here and there. And, for the casual tech blogger, this is not going to be the case. Unless you are doing a lot of writing for work, there is little time to develop your writing skills. This leads many people to turn away from writing a technical article even though they might have some of the best ideas, if only they could get them into a usable format.

Now, the next big item on why it is so hard to write technical articles is that gathering together the technical information is not easy. Don’t get me wrong, that is not to say that there are not a number of topic that you can write on. That is far from the truth. There are probably thousands of areas and topics that can be written about. But, just like writing, it takes time to gather that information. Now, there are a few high profile bloggers that are able to dedicate their jobs to writing technical blogs. Many of them are evangelist or full time employees whose job it is to talk about certain technical areas. That is great for them, and to honest, I am a bit envious. For the average person devops engineer or develop it is not so easy.

There are some companies that will allow you to blog on the work you are doing, but for most people that is not the case. So, on top of your full time job, you then most go and use your own resources and your own time to work on getting together the data, the code, and whatever else, to get together the technical information just to begin writing the article. Once the work has been done to vet the project or the topic that you are looking at writing about, you have to circle back around and figure out how it is that you want to get the information together to put it in a story for others to read. This goes back to item number one, and I have already mentioned how difficult even that first step is. Now we are taking it to a higher level by saying you do not just have to write a story, but that you must structure it around the technical information. 

So, after getting the data together and know what you are going to write about, you have to structure it. There are screen shots that have to be made, code snippets that must be shared, links to technical information that must be included. As easy as all of that sounds, it is much harder than it sounds. Figuring out how to clip an image or not make it be 4 megs in size, or how to get the images aligned so that it all looks correct. Then there is the question of where do you put your code you are going to share. How do you get the numbers showing on code? How do you enable syntax highlighting on the code samples? All of that is not easy. All of that can be overwhelming when you just want to write an article to share some information with other techies.

And the last fact of the matter is that you never know if all your efforts are for naught. You could write a great article, but if people don’t find out about it, then what do you do? Do you keep plugging away and hope people will stumble upon the articles you have written? Nobody wants to do a bunch of work for nothing. So, in recap, writing technical articles is difficult. It can be hard, and it is not always rewarding. I thank all the people that stick with it, but I understand all of those who don’t.

Making Adafruit Pixel Goggles

Let me start with the motivation for this project. On one hand, I had never really created a solution with a micro controller, and on the other I wanted to create a cool Halloween costume. The rest of my family was getting dressed up in steampunk outfits, and for a change, I wanted to add something to the mix. But, to do this, I was going to have to learn how to program this micro controller, solder, and how to get the code on the NeoPixel. I know that some people would laugh and say that this would be an easy project, but when you have never worked on this before, it is a completely different case.

Enough back story, time to walk through the build process. First, there are a the items that you will need to build out the entire unit. You will need a soldering iron. That is if you want to make it permanent. Otherwise, you will need a pair of goggles, 2 Adafruit NeoPixels, and an Adafruit Trinket. You also need a portable battery and a hookup to provide the power to the NeoPixel rings. I can break down the list and links, but they may go bad after a while. There is also a kit

There are instructions on the Adafruit website, but to be honest I found them a bit difficult to follow. There was some difficulty in understanding the instructions and what they were implying. They did have a very nice diagram that was of more use than the rest of the instructions. And, given everything, that is all you really need as far as laying out the wires.

First action is to solder on the JST-PH 2-Pin SMT Right Angle Connector to the Trinket. The USB power to the Trinket is not enough to power the NeoPixel Rings. As a result, you really need to hook up the connector so that you can provide a 5v power supply to the Trinket. Don’t make the same mistake that I did and hook up a 9v battery. This will be to much power, and the rings will only work sporadically. There is a technical reason for this, but I don’t remember electrical theory to explain it. I know enough to know that it will not work right.

Once you have the power supply soldered onto the back of the board, then the fun really begins. The power supply connectors are on the back and this is where you have to connect the adapter. There is a way to test the power supply, and that is to load the blink code onto the Trinket. I will make sure that is in my code sample. At this point, you should have a trinket that has a power supply hooked up to it, and the ability to push code to the Trinket.

That is one thing that I should have touched on. It was a huge pain to get the code to push to the Trinket when I first started working with it. I ended up have to install a very specific version of the Arduino development plaftorm, version 1.8.5, in order to get the libraries to work. In retrospect, I would have seen if I could have found a Trinket M0 board. That board is supposed to support python, and has more options when it comes to the IDE. However, as of this writing, I had to use a specific version of the Arduino IDE, but I was able to get all of it working. Just remember, that sometimes compiling the software first before pushing the button to sync makes it easier to get it all done in time.

The next thing to do was to hook up all the wires and see if it would work. Seeing as this was my first time working on the Trinket and with the NeoPixels I was not sure if I was going to get the wiring done correctly. To ensure that I was doing it right, I decided to use wires with (sic) roach clips to validate the circuit that I was going to make. At first I was intimidated by the circuit, and thought I was going to mess something up. In retrospect, it was pretty simple. 

First, lets talk about getting power to the rings. On the one side of the Trinket you have 2 connections. One connection is for outgoing power and the other is ground. There may be a technical reason for the name ground, but the simple part of it is that you need to create a complete circuit. This means the power goes from the power output on the board and goes to the power input on the NeoPixel. Then, then ground needs to come back to the board to complete the circuit. To add another ring to the mix, you take the second power connection from the NeoPixel and connect it to one of the power points on the other NeoPixel ring. Then hook up one of the ground connections to the second ground connection on the first ring. Bam! Power to both.

That is great and all, but you need to get the data to the rings. For this, the rings have a ‘data in’ and a ‘data out’ port. Unlike the power connection, this does not have to complete an entire circuit. You just select the outbound pin you want to use from the Trinket and connect it to the data in slot on the first board, then you connect the data out connection from that board to the data in on the second board. That is all. I thought there was more to it. There is not.

I will be the first to admit that I did a horrible job at soldering. It was my first time, and I had the temperature set to low, and then I had the wrong power level, so when I thought it was wrong, it turned out to be right, and I redid the entire thing. Don’t be me. Make sure you check the power 6 times and test after every solder. Unless, you are like a friend of mine. He had a job doing soldering, and he is a godlike at it.

The last part is the code. Ah, this is much more fun, and I think I might come back and revisit this later. Needless to say, the samples are here. I am a big fan of github and bitbucket for storing and sharing projects. That is a song and dance for a different day.

Micro controller Hacking coming soon

I have been hacking on my costume, and as such, have not gotten around to posting the code and pictures of the project that I am currently working on. As soon as Halloween roles by, I should be able to get that up here. Hopefully, you will enjoy the costume and the work that I have done on getting the goggles up and running. 

Normally, I would have tried to add it in chunks, but I have been learning how to solder, learning how to code this thing, and have been trying to get the rest of my costume configured. 

That being said, I am working on it. Oh, and for work I have been learning all the new nuances to python 3.6. Almost like learning another language from what python used to be. So, hold out for another few days, and I will share my code and some pics of the work as I have learned how to solder, and how to write a bit of code for an adafruit neopixel.

Configuring AWSCLI and Python on Windows

Trying and doing stuff on a Windows 10 machine has become a rather interesting experiment. It started as a place to be able to play video games and have access to a few programs that are not easily available on Linux, to a test of seeing if I could now do all the things on Windows 10 that I could do on Linux.

Turns out, I wanted to test out the Cloud Directory service from AWS. I figured the 2 easiest ways to do this would be via the AWS CLI and Python. It did not occur to me that I had neither of these installed until I opened up Cmder.exe and typed [code light=”true”]aws[/code] and then [code gutter=”false”]python[/code] and both came back with the not found.

Wait, what? Where are my programs!

So, now I need to install and configure both of these. The test is to see how easy or difficult it is to get this setup on this Windows machine. Quick list of the normal steps that I take to install the AWS CLI on most any Linux machine.

  1. Install Python
  2. Configure a virtual environment to hold my cli tools
  3. Use pip to install aws cli
  4. Configure aws cli
  5. Test that it all works

The first step in setting all of this up is to get Python installed on your machine. The AWS cli is based on Python, and as such you need to have python installed in order to use it. Now, there are some that will install the AWS cli to the root of the machine, and use the system’s globally installed Python. Due to having worked on multiple versions of Python at the same time, and projects that use different libraries, I almost always setup an Virtual Environment to run my Python programs and other sundry programs from. This way, I don’t cross contaminate my streams, and have a clearly defined idea about which versions I am using on different projects.

Installing Python

This is a relatively straightforward task. Click on the Python installer that suits your needs, download it, and follow the install prompts. I chose Python 3.6.7 because it is the version that I am already using when running some Lambda programs in AWS, and because there are some new changes in 3.7 that have broken a few other libraries. On big one is ‘async’ and ‘await’ now becoming keywords. Follow the prompts to install Python and the restart your favorite command line tool. I run bash via git, and use cmder.exe as my shell program.

Once you have it installed you should be able to run the following to verify that you have install python on your workspace. python --version This should output ‘Python 3.6.7’ or whichever version of  Python that you installed.

Setting up Virtual Environment and AWS CLI

The next part is to install the virutal environment and to then use that to install the aws cli. This should be able to be done with just a few commands, and then you should be up and running.  First we run python and install the virtual environment. Then we activate the environment and install the AWS cli. It is just a few simple commands, and you should then be up and running.

[code language="bash"]c:\ericv\dev\python -m venv p367
c:\ericv\dev\> p367\Scripts\activate.bat
(p367) c:\ericv\dev\> pip install awscli
(p367) c:\ericv\dev\> aws help[/code]

And bamm! you are done. Now, there is always configuring aws to use params, but that is another issue. But, it took me longer to write this up, than it took me to do the install.  That in of itself is a good thing to know. Now, the question is if I will run into any more problems. But, so far so good.