How do I setup a new python project?

Python prides itself on there only being one best way of doing things. However, if you have ever had the desire to create a package that can be distributed in python, then you may have run into some frustration. Out of the box, there does not appear to be a single clear concise way of creating a new project. After hearing over and over again, that the correct way in Python should be obvious, in this area, it seems that this is definitely not the case.

There should be one-- and preferably only one --obvious way to do it.

— Zen of Python

However, it does seem that there are a couple of solutions that have been put forward to solve this problem, and none of them have officially been endorsed by the people that manage Python. You can do it all by hand, and I give you credit if you do, but this is more work than I want to keep up with. I started to walk through the documentation to use setuptools to configure a package, and almost wanted to cry. Not really, but when you cannot give a clean concise way to build and distribute a python package, there is a problem.

The only thing that is clear is that you should have a setup.py file. Other than that, you are on your own. As I said before, the documentation from setuptools is laughable. Maybe if I did not mind reading pages upon pages to get a simple package started I would be OK, but I don’t have the patience for it. I thought that it would make sense to look at existing packages, but once again, everybody seems to setup their projects differently. Don’t get me wrong, there is nothing wrong with developers taking their own routes to setup a package, especially after seeing how little guidance there is. Maybe packaging was an afterthought for Python.

Having used Ruby in the past, I was sure that there had to be packages out there to assist in setting up the initial package structure and environment. Now, I do not expect it to write the code for me, but the basics of what is in the package, file layout, author information, etc. And after a bit of searching and head scratching I finally found what I was looking for. Well, at least I got a bit further down the rabbit hole. I had honestly thought that by this point I would be working on migrating my app from Ruby to Python, not trying to figure out how to create a package.

What tools are there?

It seems that upon first glance, the top three tools for creating Python packages are Poetry, Pipenv, and Hatch. And when I say creating, I mean creation and management of the packages. There is always doing it by hand, and maybe I will end up there, but that goes against the grain of automation.

I have spent a bit of time looking at these three options, plus managing it by hand. After looking at it for a bit, I think that I am going to throw Pipenv out the window. The lack of proper documentation is a stopping point for me. Also, it seems like there is a fork that is now responsible for the actual development, and not the original source itself.

That leaves me with Hatch and Poetry. Decisions, decisions, decisions. I am not sure which route I am going to go down.

Which one do you choose?

I think that I am going to start by creating my project using both methods. Early on, it should be simple enough to create the source and copy it between projects. The real question will be, which one will make management easier in the long run. That means, in my next post, I am going to start building, or more accurately, rebuilding cfmason.

Within a week or two I should be able to make my decision. But, I am going to build them out from scratch both ways, and record the process. Heck, that will probably be harder than the coding itself. Documenting this stuff is not easy.

See you in a week.

How do I get rid of the bell in Vim on PyCharm on Windows?

One of the most infuriating things in the world is when the the bell goes off continuously when using the Vim plugin in PyCharm. Actually, this holds true for all of the JetBrains IDEs, but I have been working mostly with Python recently, and as such, it is PyCharm that is on the top of my mind.

While there are a number of plugins that you can get for the JetBrains products, the one that I always end up installing is Vim. Call me old school, but back in the day, I had to work on remote systems. It was either Vim or Pico, and I chose Vim. Now that I have modern IDEs, I still end up using Vim for all my editing needs. The muscle memory is built in, and I don’t even think about it anymore.

But what about the Bell?

While I have used Linux or Mac as my primary development environment for the last 10 years, I am recently decided to see how well I can survive while coding using Windows. That being what it is, I installed PyCharm, got the Vim plugin installed, and went to get to work. And then it happened, the audible bell of hell. I added my regular vimrc file into my home directory, and no dice.

The bell was still there.

I double checked my settings, and to no avail, the stupid bell would not go away. It was driving me crazy. I turned off the volume on my computer while I was working on the issue. And then, after digging through the documentation on the JetBrains documentation website I found the answer. The Vim modules for JetBrains does not look at the .vimrc file.

The fix….

The Vim plugin for PyCharm and all the JetBrain products uses the .ideavimrc file.

set noeb
set novb
set belloff=all

Then, all you have to do is save and exit. Restart the IDE, and you are done.

No more bell. Well, at least on Windows. But, this should not be that hard, but then again, I think with some of the settings that are built into the various IDEs, they did not want to pick up all the settings in a normal .vimrc file. I may dig more into that in the future, but the next step is to see how well Python plays on Windows compared with on Linux/Mac.

An Introduction to AWS Step Functions – Part 2

In Part 1, I went through a whirlwind tour and left example code on how to create a Lambda function and a Step Function. The thing is, I did not explain how the code works. That is what this article is about. I am going to be using the same code from the previous post, but will be going through the code and explaining it. We shall see how well the formatting works.

Jumping right into it. This is the entire code for the StepFunction from the previous post. As of right now, this code has to be json. I did not find any information about supporting yaml. Here is the documentation provided by AWS. It is pretty thorough, but it could be a bit clearer. My explanation could be better or worse.

{
  "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
  "StartAt": "LambdaFunction",
  "States": {
    "LambdaFunction": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:aws-serverless-repository-hello-w-helloworldpython-1JQ8TEEDUAHCE",
      "ResultPath": "$.taskresult",
      "Retry": [
        {
          "ErrorEquals": ["CustomError"],
          "IntervalSeconds": 1,
          "MaxAttempts": 2,
          "BackoffRate": 2.0
        },
        {
          "ErrorEquals": ["States.TaskFailed"],
          "IntervalSeconds": 30,
          "MaxAttempts": 2,
          "BackoffRate": 2.0
        },
        {
          "ErrorEquals": ["States.ALL"],
          "IntervalSeconds": 5,
          "MaxAttempts": 5,
          "BackoffRate": 2.0
        }
      ],
      "Next": "ChoiceState"
    },
    "ChoiceState": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.taskresult.value1",
          "StringEquals": "value1",
          "Next": "SuccessState"
        },
        {
          "Variable": "$.taskresult.count",
          "NumericLessThan": 5,
          "Next": "LambdaFunction"
        }
      ],
      "Default": "FailState"
    },
    "SuccessState": {
      "Type": "Succeed"
    },
    "FailState": {
      "Type": "Fail",
      "Cause": "Invalid response.",
      "Error": "ErrorA"
    }
  }
}

Looking at this for the first time, it all seems a bit overwhelming. However, after breaking it down, you will see that while you have to be precise with it, Step Functions are fairly easy to read. (Writing them is still a PITA, but that is just what it is) When you get down to it, the Step Function configuration file only contains 3 sections: ‘Comment‘, ‘StartsAt‘, and ‘States‘. That is it.

Let us take a quick look at the the first two mentioned above, Comment and StartsAt. The former of these is just a comment so you know what the purpose of the Step Function is. On the other hand, you could use this for whatever comment you like. Then, there is StartsAt. StartsAt, actually links to a State. And the States section is what contains all the real meat of the Step Function. The value that is used for StartsAt is the name of one of the States.

States

States are where everything really happens when it comes to Step Functions. They are the individual units that when strung together, make the magic happen. When you look at it from this perspective, it sounds ridiculously easy. But, just like everything else, the devil is in the details. The key to understanding States is to know what the various components that make up a State, and the type of states that are available.

Since this is an introduction, and based off of my code here, I am not going to go through all the options. However, there are docs from Amazon on it. The think about it is, it is not simple. Based on the docs, the only required field is Type. Great, but what are the various types? How do I link them together to make a cohesive whole? What are these other options and how do I pass variables from one section to the next and use them over and over again in a loop? This information is available, but it is scatter all over the documentation and you have to piecemeal it all together to figure out what is available.

What are the State types?

  • Pass – used to take input and pass to output. Basically, used to debug step functions, so you know what the heck is happening between States.
  • Task – this is where the work gets done. It can be a Lambda function or any other supported service. This will take your input and from there provide output to another State to be consumed
  • Choice – handles branching in your Step Function by looking at the output from the previous State and sending the Step Function to perform the next decided upon State in the workflow.
  • Wait – allows you to put a pause in your Step Function to wait for an action to finish, like deploying CloudFormation or some other long running task.
  • Succeed – the state machine has finished with success. This one is actually simple and does just what it says.
  • Fail – the end state where all has not gone according to plan. This is the opposite of Succeed, and allows for some reporting on what went wrong.
  • Parallel – run States at the same time. I have not yet worked with this, so I do not have as much information as I would like on this type of State

In the example above, I use only 4 of the States from above, Task, Choice, Succeed, and Fail. Mainly because I wanted to touch on a good bit of the basics without going to far down the rabbit hole. For my example, my Task was a Lambda function. To me this is the most logic way to use step functions, but there are probably many more that I have not considered. I also wanted to employ a loop. This being that I wanted to test flow control, and passing variable to and from Lambda functions. I could have strung the Lambda functions together in a straight line, but I wanted to test out branching as well.

The big factor to remember is that the order of the States does not matter. What matters is linking which State comes next. You could use a single Choice State to manage the flow through the entire Step Function, and based upon the input move onto any number of other States or back to a previous one.

[code]
],
“Next”: “ChoiceState”
},
[/code]

This little block of code is a great example. The Next keyword is used, and the following word is the name of the next State block that is going to be executed. The name of each State is identified by the start of the json template block. In the example above, I am not going to a State of type Choice, but have created a State that is called ChoiceState. In retrospect, much of this could have been done cleaner, but it was an example that I put together rather quickly. This is the declaration:

[code]
},
“ChoiceState”: {
“Type”: “Choice”,
“Choices”: [
[/code]

I am going to end here for today. The next section is going to be about passing data from the Step Function to Lambda, and how to reuse it. But, I want to work on a smaller block of code and focus on just the one thing.

Getting Started with AWS Step Functions Part I

AWS came out with Step Functions a few years ago, and up until recently, I have not had the opportunity to dive in and give them a try. Yes, I could build my own pipeline or state machine, but the idea behind Step Functions is that it does most of the heavy lifting for you. That, and it ties into other AWS services. As such, I decided to dive into getting started, and looked at the demo options and walkthroughs that were available. None of them met my needs, so I rolled my own.

The idea is to see how I can create a Step Function that will run multiple loops, and call a Lambda function multiple times. What I wanted to test was the following:

  • Pass Variables into the Step Function and see how they are handled
  • Call a Lambda function multiple times
  • Create a loop using the Step Function DSL
  • Test output from Lambda and make a decision based upon it
  • Figure out any gotchas and how to trigger Step Functions

Let’s dive in. Now, this is the final result. It took me a few iterations to actually get to this point. Smarter people than I might be able to get it done on one go, but not I.

Lambda Code

I came up with a simple Lambda function written in Python 3.6. All that I wanted to do was to perform a loop with Step Functions, and then get output the values. Simple. And as you can see, this code is pretty simple. It could be streamlined, but it was quick and easy to write.

def lambda_handler(event, context):
    print('value1 = ' + event['key1'])
    print('value2 = ' + event['key2'])
    print('value3 = ' + event['key3'])
    taskresult = event.get('taskresult', None)
    if taskresult is None:
        count = 0
    else:
        count = taskresult.get('count', None)
    
    if count is None:
        count = 0
    else:
        count = count +1
        
    if count < 5:
        output = {
            'count' : count,
            'value1' : 'ThereIsNoSpoon';
        }
    else:
        output = { 'value1' : event['key1'],
                    'value2': event['key2'],
                    'count' : count }
    return output

Now we need to move onto the body of what we are working on, and that would be the Step Function. Step Functions have their own language or domain specific language (DSL) that is used to define the state machine. I wanted more than just a “Hello World” example. The idea was to loop through a step functions. Make sure that I could call it multiple times, and then either go to a success or failed state

AWS Step Function Code

{
  "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
  "StartAt": "LambdaFunction",
  "States": {
    "LambdaFunction": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:aws-serverless-repository-hello-w-helloworldpython-1JQ8TEEDUAHCE",
      "ResultPath": "$.taskresult",
      "Retry": [
        {
          "ErrorEquals": ["CustomError"],
          "IntervalSeconds": 1,
          "MaxAttempts": 2,
          "BackoffRate": 2.0
        },
        {
          "ErrorEquals": ["States.TaskFailed"],
          "IntervalSeconds": 30,
          "MaxAttempts": 2,
          "BackoffRate": 2.0
        },
        {
          "ErrorEquals": ["States.ALL"],
          "IntervalSeconds": 5,
          "MaxAttempts": 5,
          "BackoffRate": 2.0
        }
      ],
      "Next": "ChoiceState"
    },
    "ChoiceState": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.taskresult.value1",
          "StringEquals": "value1",
          "Next": "SuccessState"
        },
        {
          "Variable": "$.taskresult.count",
          "NumericLessThan": 5,
          "Next": "LambdaFunction"
        }
      ],
      "Default": "FailState"
    },
    "SuccessState": {
      "Type": "Succeed"
    },
    "FailState": {
      "Type": "Fail",
      "Cause": "Invalid response.",
      "Error": "ErrorA"
    }
  }
}

The way that this code works is as follows. Everything works around States. So, you have to move from State to State. This is a key concept when it comes to Step Functions. Now, there are multiple State types, but I am not going to go into that now. The key factor is that you will go through and loops if a proper return value is not returned. Looking at it now, it looks like a bunch of gobbledygook. I am going to have to come back and write up how this works later.

This is what the visual representation looks like when viewed in the AWS Step Function page. There is a defined ‘Start’ and ‘Stop’. The other stages match what was named in the previous section. The code works to present a model that you can follow.

The cool think about AWS Step Functions is that they guarantee a run. And in a situation where you need to ensure that the code is run, and you need a guarantee. This is mostly due to the cost that is associated with it. Running Lambda that Triggers on SQS would be cheaper, but not as easy to ensure.

Back on with our stuff now. We are looking at how we execute the AWS Step Function. Now we need to execute it. Right now, I am not going to go into the logic around passing variables around. Needless, you will need to understand that when writing your own, and I am going to have to revisit it.

Execution. A couple of items to note.

  • Each execution has to have a unique name.
    • Note, this will bite you when you are testing, and think about this when executing it via automation.
  • It takes in an action just like Lambda, via json
  • Making a small change in the inputs can cause madness
{
  "key1": "value1",
  "key2": "value2",
  "key3": "value3"
}

The output will also be in json, and you can see the results in the visual display.

{
  "key1": "value1",
  "key2": "value2",
  "key3": "value3",
  "taskresult": {
    "value1": "value1",
    "value2": "value2",
    "count": 5
  }
}

This is what the output looks like.

I will go into more detail on the breakdown of the Step Function in the next post. There is a lot to be covered, and this just scratches at the surface.

Writing Tech Blogs are Hard Work

I once read an article that said more people need to write technical blogs. That the problem with much of the technology field was that people did not write in-depth articles on the stuff that they are doing. And, if more people were able to take the time and post a blog entry here and there we would all be better for it. As nice as that sounds, I have to say that writing technical posts are difficult, time consuming, and can quickly go out of date.

First off, writing in general is not easy, and that is before you get to adding the technical part on top of it. There are a number of brilliant developers, system engineers, and devops people that can create some of the most complex and unique solutions to problems, and yet cannot begin to write the first bit of documentation or narratives to describe what it is that they have done. It is not that they are dumb, they just have not honed the writing skill, or just might not be proficient at it. Writing is like coding, if you don’t use it, you can lose it. Also, it is a skill that can be developed and honed overtime. Myself, my writing skills are rusty after taking a long time off from it, and I am trying to get back into the flow.

There are some people that recommend to become talented at writing, you need to be able to dedicate an hour a day to it, or 5 hours a week. And that is just the writing part. That does not tie in working on the technical elements that are needed to provide content for the audience you are trying to reach. Maybe a few years back when I did not have family obligations this would have been possible, but now I have to sneak in time here and there. And, for the casual tech blogger, this is not going to be the case. Unless you are doing a lot of writing for work, there is little time to develop your writing skills. This leads many people to turn away from writing a technical article even though they might have some of the best ideas, if only they could get them into a usable format.

Now, the next big item on why it is so hard to write technical articles is that gathering together the technical information is not easy. Don’t get me wrong, that is not to say that there are not a number of topic that you can write on. That is far from the truth. There are probably thousands of areas and topics that can be written about. But, just like writing, it takes time to gather that information. Now, there are a few high profile bloggers that are able to dedicate their jobs to writing technical blogs. Many of them are evangelist or full time employees whose job it is to talk about certain technical areas. That is great for them, and to honest, I am a bit envious. For the average person devops engineer or develop it is not so easy.

There are some companies that will allow you to blog on the work you are doing, but for most people that is not the case. So, on top of your full time job, you then most go and use your own resources and your own time to work on getting together the data, the code, and whatever else, to get together the technical information just to begin writing the article. Once the work has been done to vet the project or the topic that you are looking at writing about, you have to circle back around and figure out how it is that you want to get the information together to put it in a story for others to read. This goes back to item number one, and I have already mentioned how difficult even that first step is. Now we are taking it to a higher level by saying you do not just have to write a story, but that you must structure it around the technical information. 

So, after getting the data together and know what you are going to write about, you have to structure it. There are screen shots that have to be made, code snippets that must be shared, links to technical information that must be included. As easy as all of that sounds, it is much harder than it sounds. Figuring out how to clip an image or not make it be 4 megs in size, or how to get the images aligned so that it all looks correct. Then there is the question of where do you put your code you are going to share. How do you get the numbers showing on code? How do you enable syntax highlighting on the code samples? All of that is not easy. All of that can be overwhelming when you just want to write an article to share some information with other techies.

And the last fact of the matter is that you never know if all your efforts are for naught. You could write a great article, but if people don’t find out about it, then what do you do? Do you keep plugging away and hope people will stumble upon the articles you have written? Nobody wants to do a bunch of work for nothing. So, in recap, writing technical articles is difficult. It can be hard, and it is not always rewarding. I thank all the people that stick with it, but I understand all of those who don’t.