Certbot Cheat Sheet

Using DNS for challenge

certbot -d foo.bar.com --manual --prefered-challenges dns certonly

Apache Cert Config

SSLCertificateFile /etc/letsencrypt/live/foo.bar.com/fullchain.pem
SSLCertificateKeyFile   /etc/letsencrypt/live/foo.bar.com/privkey.pem

Clean up Docker artifacts in dev environments

Remote dev environments that get continuous Docker deployments should have their images and volumes cleaned up regularly.

Pass the following commands over SSH to the remote server, perhaps in a Jenkins job for example:

 sudo -- bash -c 'docker volume rm $(docker volume ls -f dangling=true -q)' > /dev/null 2>&1  
 sudo -- bash -c 'docker images --quiet --filter=dangling=true | xargs --no-run-if-empty docker rmi' > /dev/null 2>&1  
 echo $?  

echo $?  .. is used at the end because if the images are already cleaned up there may be an exit status code indicating and error .. which can actually safely be ignored so echo $? will evaluate it so that Jenkins will not report the job as failed.

Connect JasperServer to a Postgres Data Source with SSL

In response to question such this:

• https://community.jaspersoft.com/ireport-designer/issues/4135

In the JDBC connection string, just add this:

?ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory

i.e. just setting ssl=true is not enough.

And just to state the obvious, your Postgres server must support SSL connections.

The bait and switch of open source

Great presentation by Katrina Owen. You need to sign up for a Safari Books account but (at this time) its free and does not require a credit card. If you don't want to give them your email, just use mailinator or something.

https://www.safaribooksonline.com/library/view/oscon-2016-/9781491958476/video284487.html


Some good points from this talk:

• Understand the the difference of "issues" vs "symptoms" in your product
• When explaining your product, talk about it's benefits, not its features
• "Manage your energy rather than your time"

Boilerplate Java for AWS Lambda invoked from AWS API Gateway

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import java.util.HashMap;
import java.util.Map;

public class LambdaAPIGateway {

    public Map<String, Object> handleRequest(Map<String, Object> request, Context context) {
        LambdaLogger logger = context.getLogger();
        logger.log("Function version: " + context.getFunctionVersion() + "; ");
        logger.log("Event: " + request.toString());
        String bodyAsJson = "{\"data\":\"ok\"}";
        Map<String, Object> result = Respond(200, bodyAsJson, context);
        return result;
    }

    public static Map<String, Object> Respond(int httpStatus, String bodyAsJson, Context context)  {
        Map<String, Object> retval = new HashMap<>();
        /**
         Response MUST be in a specific format with "headers"; "statusCode" and "body" ONLY
         Example: { "headers": {"Content-Type":"application/json"}, "body":"...", "statusCode":200 }
         See http://amzn.to/2lY4oQB
         (under the heading Output Format of a Lambda Function for Proxy Integration)
         */
        Map<String, Object> headers = new HashMap<>();
        headers.put("Content-Type", "application/json");
        headers.put("x-request-id", context.getAwsRequestId());
        retval.put("headers", headers);
        retval.put("statusCode", httpStatus);
        retval.put("body", bodyAsJson);
        return retval;
    }
}

The AWS documentation provides a Java example which uses inputStream and outputStream and takes a lot more code: http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-as-simple-proxy-for-lambda.html#api-gateway-proxy-integration-lambda-function-java

I like this version better because is simpler and shorter.

More comprehensive example here:
https://github.com/sylnsr/boilerplate-code/blob/master/java/aws/LambdaAPIGateway.java

Using Prometheus with Java in a Jersey project

Step 1, add the dependency to your project. If you're using maven, add the dependency:

<dependency>
    <groupId>io.prometheus</groupId>
    <artifactId>simpleclient_servlet</artifactId>
    <version>0.0.6</version>
</dependency>

Step 2, create a class and register some metrics. Here I have a summary and some counters:

public class Metrics {

    public static final Summary requestLatency = Summary.build()
            .name("requests_latency_seconds")
            .help("Request latency in seconds.").register();

    public static final Counter requestFailures = Counter.build()
            .name("requests_failures_total")
            .help("Request failures.").register();

    public static final Counter requestsTotal = Counter.build()
            .name("requests_total")
            .help("Request.").register();

    public static final Counter uploadedFilesSucceeded = Counter.build()
            .name("upload_file_success")
            .help("Total files uploaded to S3.").register();

    public static final Counter tmpFilesNotCleared = Counter.build()
            .name("temp_files_not_cleared")
            .help("Total files that could not be removed from cache").register();

}

Step 3, set a class level variable in your main class to instantiate your Metrics class:

private final Metrics metrics = new Metrics();

Step 4, manipulate your metrics in your code as needed. For example you need to call the inc() function on your counters to increment them. You can add to the requestLatency metric as follows:

@GET
@Path("/something")
@Produces(MediaType.TEXT_PLAIN)
public String Something() {
    try {
        Summary.Timer timer = Metrics.requestLatency.startTimer();
        // do some work ...
        timer.observeDuration();
    } catch (Exception e) {
        return e.getMessage();
    }
    return "It works!";
}

Step 5, add a web method to dish out your metrics. This is the part that I did not like from other online examples (e.g. http://www.boxever.com/easy-java-instrumentation-with-prometheus) ... since they use the Metrics Servlet provided by Prometheus and all that does is user the writer to gather the data for you, so you need to mess with registering and deploying a new servlet within your code. TOO MUCH HASSLE. Just do what the Prometheus servlet does. Load the registry and use the TextFormat class to get your metrics. No need to deploy a servlet! :

@GET
@Path("/metrics")
@Produces(MediaType.TEXT_PLAIN)
public String Metrics() {
    StringWriter writer = new StringWriter();
    try {
        io.prometheus.client.exporter.common.TextFormat.write004(
                writer, CollectorRegistry.defaultRegistry.metricFamilySamples());
    } catch (Exception e) {
        return e.getMessage();
    }
    return writer.toString();
}

Done!


Here is some sample output from one of our apps using this technique:

# HELP requests_failures_total Request failures.
# TYPE requests_failures_total counter
requests_failures_total 0.0
# HELP temp_files_not_cleared Total files that could not be removed from cache
# TYPE temp_files_not_cleared counter
temp_files_not_cleared 0.0
# HELP requests_latency_seconds Request latency in seconds.
# TYPE requests_latency_seconds summary
requests_latency_seconds_count 0.25
requests_latency_seconds_sum 1.0
# HELP upload_file_success Total file uploaded to S3.
# TYPE upload_file_success counter
upload_file_success 25874588.0
# HELP requests_failures_total Request failures.
# TYPE requests_failures_total counter
requests_failures_total 2.0

Postgresql upsert example with CTE (before upsert support from 9.5)

 with heartbeat_data (source, time) as (  
  values ('test', now())  
 ),  
 update_query as (  
  update dvs_system.heartbeats  
    set last_beat = heartbeat_data.time  
  from heartbeat_data  
  where source_key = heartbeat_data.source  
  returning true as updated  
 )   
 insert into dvs_system.heartbeats (source_key, last_beat)  
  select source, time  
  from heartbeat_data  
  where not exists (  
   select 1 from update_query where updated = TRUE  
  )  
 ;  

Reference: http://stackoverflow.com/questions/1109061/insert-on-duplicate-update-in-postgresql/8702291#8702291

Postgres: Create a function to create a new logging table which inherits from another

When creating large logging tables, it better to define the structure of the logging table then create tables that inherit from that table, to put the data into. It makes look-ups on the table faster if you are searching for info in your logs relevant to data in one partitioned child table and it also makes it easier to trim the logs since you can just drop a child table of logs when the time is appropriate.

See http://www.postgresql.org/docs/9.1/static/ddl-partitioning.html

 
 
 CREATE OR REPLACE FUNCTION app_system.create_request_log_partition(_date timestamp without time zone)  
  RETURNS void AS  
  $BODY$  
 DECLARE  
      _table text := format('request_log_%s_q%s', date_part('year', _date), date_part('quarter', _date));  
 BEGIN  
        
      EXECUTE  
            'CREATE TABLE system.'|| _table ||'() INHERITS (system.access_log);'||E'\n'  
           ||'CREATE INDEX '|| _table ||'_created_idx ON system.'|| _table ||'(created_at);'||E'\n'  
           ||'CREATE INDEX '|| _table ||'_api_request_idx ON system.'|| _table ||'(api_request);'||E'\n'  
           ||'GRANT INSERT, SELECT ON system.'|| _table ||' TO myapp;'||E'\n'  
           ;  
 END;  
 $BODY$  
 LANGUAGE plpgsql VOLATILE  
 COST 100;  
 ALTER FUNCTION app_system.create_request_log_partition(timestamp without time zone) OWNER TO postgres;  
   

Generating self-signed SSL cert and PEM

Sign in as root then run this

cd /etc/ssl
openssl req -x509 -new -nodes -newkey rsa:2048 -keyout haproxy.key -out haproxy.crt
cat ./haproxy.crt ./haproxy.key > ./haproxy.pem

You can  name the files whatever you want instead of using "haproxy", which I use since I like to have SSL termination in my haproxy server.

Injection GIT branch / tag name into config.properties file

Prerequisite: In your project you need the file src/main/resources/config.properties

Of course you can adapt to using whatever file you want.

Note that this assumes you're using a "version=" property in your config.properties file


Step 1: Add the maven plugin to run an executable

1:  #!/usr/bin/env bash  
2:    
3:  # Get the git branch / tag name  
4:    
5:  # first, see of we have branch information (this will not be available if we checked out a tag)  
6:  TAG_OR_BRANCH="$(git rev-parse --abbrev-ref HEAD | egrep -o '([0-9]{1,}\.)+[0-9]{1,}')"  
7:    
8:  if [ "$TAG_OR_BRANCH" == "" ]; then  
9:    
10:    # if we can't get branch info, then we must be in a tag, so use that  
11:    TAG_OR_BRANCH="$(git describe | egrep -o '([0-9]{1,}\.)+[0-9]{1,}+(-b[0-9]{1,})')"  
12:  fi  
13:    
14:  # Remove the version from the properties file  
15:  sed -i '/^version=/ d' src/main/resources/static/version  
16:    
17:  # Add the version to the properties file  
18:  echo "Writing version $TAG_OR_BRANCH"  
19:  echo "version=$TAG_OR_BRANCH" >> src/main/resources/static/version  

Generate Golang struct (model) from Postgres tables

A handy Postgres SQL statement to generate Golang models (structs) for you.

WITH models AS (
  WITH data AS (
    SELECT
        replace(initcap(table_name::text), '_', '') table_name,
        replace(initcap(column_name::text), '_', '') column_name,
        CASE data_type
        WHEN 'timestamp without time zone' THEN 'time.Time'
        WHEN 'timestamp with time zone' THEN 'time.Time'
        WHEN 'boolean' THEN 'bool'
        -- add your own type converters as needed or it will default to 'string'
        ELSE 'string'
        END AS type_info,
        '`json:"' || column_name ||'"`' AS annotation    
    FROM information_schema.columns
    WHERE table_schema IN ('dvs_app', 'dvs_system')
    ORDER BY table_schema, table_name, ordinal_position
  )
    SELECT table_name, STRING_AGG(E'\t' || column_name || E'\t' || type_info || E'\t' || annotation, E'\n') fields
    FROM data
    GROUP BY table_name
)
SELECT 'type ' || table_name || E' struct {\n' || fields || E'\n}' models
FROM models ORDER BY 1

Handy PowerShell Scripts

(because I don't want to add another GitHub repo)

Invert mouse scrolling (from http://superuser.com/questions/310681/inverting-direction-of-mouse-scroll-wheel)

Get-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Enum\HID\*\*\Device` Parameters FlipFlopWheel -EA 0 | ForEach-Object { Set-ItemProperty $_.PSPath FlipFlopWheel 1 }

Anarchic Development Efforts

Nokia, once the leader in mobile technologies really messed up big. They were poised (perhaps) to be where Apple is today with the iPad and the iPhone however they had various deeply seeded problems which ultimately caused the demise of their R&D efforts. This interesting article is a great read for technology project managers. One of the most interesting things I found was the mention of "anarchic development efforts" at Nokia which was a major factor leading to ultimate project failure. By contrast the leadership culture at Apple has been very disciplined and focused under the collaborative-dictatorship of Steve Jobs. On a personal note, this is why, when I am heading up development teams, I have no time for "anarchic development efforts". Developers that don't get with the program must be let go or the entire project will be in jeopardy. I think this article highlights a sterling example of issue.

Installing Linux 64bit driver for eToken USB

If needed install, opensc and pcsc-lite

yum install opensc pcsc-lite

Download the driver, unzip and install

wget http://download.fh-swf.de/dvz/software/eToken/Linux/Software/SafeNetAuthenticationClient_Linux_8_1.1.zip unzip SafeNetAuthenticationClient_Linux_8_1.1.zip cd SAC\ 8.1\ Linux/x86_64/ unzip SAC_8_1_0_4_Linux_RPM_64.zip cd SAC_8_1_0_4_Linux_RPM_64/Signed\ installation\ scripts/ chmod +x signed-install_SafenetAuthenticationClient-8.1.0-4.x86_64.rpm.sh cp ../RPM/SafenetAuthenticationClient-8.1.0-4.x86_64.rpm . ./signed-install_SafenetAuthenticationClient-8.1.0-4.x86_64.rpm.sh

You should see the following (choose 1 for English):

Searching SafenetAuthenticationClient-8.1.0-4.x86_64.rpm... OK Searching RPM-GPG-KEY-SafenetAuthenticationClient... OK Deleting existing key Importing key: RPM-GPG-KEY-SafenetAuthenticationClient Starting installation ########################################### [100%] ########################################### [100%] Adding Token security provider......done. Starting PC/SC smart card daemon (pcscd): [  OK  ] SafeNet Authentication Client installation completed. Choose Client Language: ------------------------ 1. English.....(En) 2. Spanish.....(Es) 3. French......(Fr) 4. Italian.....(It) 5. Japanese....(Jp) 6. Korean......(Ko) 7. Russian.....(Ru) 8. Chinese.....(Zh) 9. Portoguese..(Pt) 10. Thai.......(Th) Your choice [1-10] -->1 Installing Language code En. Done!

Check

opensc-tool --list-readers

The result should be similar to:

# Detected readers (pcsc) Nr.  Card  Features  Name 0    Yes             AKS ifdh 00 00 Find out which slot the device is in: pkcs11-tool --module /usr/lib64/libeToken.so -L You should see something to the effect of: Available slots: Slot 0 (0x0): AKS xxxx 00 00 token label: my label token manuf: SafeNet, Inc. token model: eToken token flags: rng, login required, PIN initialized, token initialized, other flags=0x200 serial num : xxxxxxx Slot 1 (0x1): (empty) Slot 2 (0x2): (empty) Slot 3 (0x3): (empty) Slot 4 (0x4): (empty) Slot 5 (0x5): (empty)       ----------------- Acknowledgements:  "user4668" for showing how to see the used slots http://bit.ly/1w2dUT8   Other Sources for the SAC: http://www.digicert.com/util/SafeNetAuthenticationClient-610-011815-002_SAC_Linux_v8.1.zip http://www.digicert.com/util/SafeNetAuthenticationClient.8.2.27.0.dmg

Installing MS fonts into Centos for converting DOCX files to PDF using DOCX4J

a)  Install MS TT Fonts using information here:

http://linuxg.net/how-to-install-the-microsoft-core-fonts-on-fedora-18-and-19/

b)  If you need font mapping you can setup a font.properties file. I copied the one on my Mac, from
 /Library/Java/Home/lib/fontconfig.properties.src
to
 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/lib/font.properties
.. on my Centos development box .

Notes:

After installing the MS fonts, it must add a file to the JDK lib as (the path will vary based on whatever JVM you are using):

/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/lib/msttcorefonts-2.5-1.spec

The reason for renaming fontconfig.properties.src to font.properties, is found here:
http://docs.oracle.com/javase/8/docs/technotes/guides/intl/fontconfig.html

Remember to restart any Java app that's running and needs to find the newly installed fonts.

Generating certs for testing PDF signing

Run the following as root:

openssl genrsa -out ca.key 2048
openssl req -new -key ca.key -out ca.csr
openssl x509 -req -days 365 -in ca.csr -signkey ca.key -out ca.crt
openssl pkcs12 -export -in ca.crt -inkey ca.key -name ca -passout pass:password -out ca.pfx

Good to know: PostgreSQL records to single JSON string

To convert a set of results to a single JSON string, use:

SELECT
ARRAY_TO_JSON(ARRAY_AGG(ROW_TO_JSON(ROW)))
as DAT FROM (
     [ your query here ]
) ROW;

Tip: In the query, if you specify field names or table.*, you will not get an object hence the JSON will not have your schema sub grouped by object name. i.e.

SELECT
ARRAY_TO_JSON(ARRAY_AGG(ROW_TO_JSON(ROW)))
as DAT FROM (
     SELECT mytable WHERE [ ... ]
) ROW;

 ... is not the same as ...

SELECT
ARRAY_TO_JSON(ARRAY_AGG(ROW_TO_JSON(ROW)))
as DAT FROM (
     SELECT mytable.* WHERE [ ... ]
) ROW;



Kudos: http://hashrocket.com/blog/posts/faster-json-generation-with-postgresql

Script to remove jpg files recursively

I am using a webcam which uploads files via FTP. Unfortunately the cam which uploads the files does not clean up old files. It's actually a set of cameras which all write file to their own directory structures. So when I need a script to clean up the old jpg files it needs to run recursively:

 #!/usr/bin/env python  
 import re  
 import os  
 __dir__ = os.path.dirname(os.path.realpath(__file__))  
 def cleanup_recursive(_dir=__dir__):  
   print "Cleaning " + _dir  
   for _item in os.listdir(_dir):  
     _path = _dir + "/" + _item  
     if os.path.isfile(_path) and re.match(".+\.jpg$", _path):  
       os.remove(_path)  
     elif os.path.isdir(_path):  
       cleanup_recursive(_dir)  
 cleanup_recursive()  

Remove Bitdefender from OSX (App Store version)

Bitdefender for Windows is a highly rated application, but I can tell you from personal experience, on OSX, it's pretty much garbage. Furthermore, the App Store version does not come with an uninstaller the same way the download (from their site) version does. After following their recommended way of removing the app you may want to remove other files also. Remove this folder: ~/Library/Application\ Support/Bitdefender\ Virus\ Scanner And this file: ~/Library/Saved\ Application\ State/com.bitdefender.BitdefenderVirusScanner.savedState The issue that I, and other nay sayers on the App Store, have with BD AV is that it completely locks up your system when running a full scan, essentially making it about as bad as some viruses. I'm using ClamXAV now instead.

PostgreSQL:: Find functions containing

Sometimes when you need to find a function which contains specific text in the definition or description, it's handy to have a function that makes that possible.

Here is a function that will allow you to do just that:

 /**  
 DEFINE  
  */  
 CREATE OR REPLACE FUNCTION my_admin_schema.sel_function_containing(_schema_name text, _search_value text)  
  RETURNS TABLE (  
   schema text, sproc_name text, arg_names text, return_type text, description text, definition text  
  ) AS  
  $BODY$  
 BEGIN  
  RETURN QUERY   
   WITH funcs AS (  
     SELECT  
       n.nspname::text AS schema  
      ,proname::text AS sproc_name  
      ,proargnames::text AS arg_names  
      ,t.typname::text AS return_type  
      ,d.description  
      ,pg_get_functiondef(p.oid) as definition  
     FROM pg_proc p  
      JOIN pg_type t on p.prorettype = t.oid  
      JOIN pg_description d on p.oid = d.objoid  
      JOIN pg_namespace n on n.oid = p.pronamespace  
     WHERE n.nspname = _schema_name  
   )  
   SELECT *  
   FROM funcs  
   WHERE funcs.definition ~* _search_value  
     OR funcs.description ~* _search_value  
   ;  
 END; $BODY$ LANGUAGE plpgsql STABLE COST 100  
 ;;  
 /**  
 EXAMPLE USAGE  
  */  
 SELECT * FROM offerpoint_admin1.sel_function_containing('my_app_schema', 'status')  
 ;;