Powershell Tips using Jenkins

Created 10 April 2019 07:13, updated 11 April 2019 08:18

At IUA we use Jenkins for CI, and we use Powershell for automation of our builds and deployments. We've always invoked Powershell using batch commands, but I recent implemented the Powershell plugin which makes life a little easier. Some tips on implementation:

Include a file

Prefix the file name with a ". "

. C:\Temp\test.ps1

Use environment variables

When running in the context of Jenkins, you can get and set environment variables. This can be useful when wanting to access a common function used across multiple scripts. For example, if you have many files wanting to use a version number, but the generation of that version number is complex enough to put it in a function, you could move that function to a common file and then store the generated value in an environment variable:

. C:\Temp\version.ps1
$env:version = GetVersion
Write-Host $env:version

Run jobs via the Jenkins API

I got this code from Stack overflow (https://stackoverflow.com/questions/46131903/why-is-powershell-not-able-to-send-proper-crumb), but adapted it to include build parameters:

function RunJenkinsJob
{
    param(
        [string]$userName,
        [string]$password,
        [string]$apiUrl,
        [string]$apiPort,
        [string]$jobName,
        [string]$jobParams      # url-encoded string of parameters e.g. myparam1=val1&myparam2=anotherval
    )

    $h = @{}
    $h.Add('Authorization', 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes("$(${userName}):$(${password})")))

    $params = @{uri = "${apiUrl}:${apiPort}/crumbIssuer/api/json";
            Method = 'Get';
            Headers = $h;}
    $apiCrumb = Invoke-RestMethod @params

    $uri = "${apiUrl}:${apiPort}/job/${jobName}"
    if ([String]::IsNullOrWhiteSpace($jobParams)) {
        $uri = "$uri/build"
    }
    else {
        $uri = "$uri/buildWithParameters?$jobParams"
    }

    $h.Add('Jenkins-Crumb', $apiCrumb.crumb)
    $params['uri'] = $uri
    $params['Method'] = 'Post'
    $params['Headers'] = $h

    Invoke-RestMethod @params

}

RunJenkinsJob -userName "jenkins" -password "mypassword" -apiUrl "http://buildserver" -apiPort "8080" -jobName "JobWithParams" -jobParams "PARAM1=value1&PARAM2=value2"
RunJenkinsJob -userName "jenkins" -password "mypassword" -apiUrl "http://buildserver" -apiPort "8080" -jobName "JobWithoutParams" 

Matching IIS W3SVC folders to web sites

Created 11 December 2018 17:07

For every web site created in IIS, a new W3SVCX folder is created in the logging folder configured for IIS. Knowing which folder applies to which site is not obvious, particularly when you have multiple sites serving the same content and it's not easy to determine this information from inspecting the log files.

The configuration can be worked out by opening the applicationHost.Config file in the %WinDir%\System32\Inetsrv\Config folder.

Under the sites element, each site is then listed with a unique id, for example:

<site name="mysite.mydomain.co.za" id="6" serverAutoStart="true">

In this case, the log files for this site will be stored in the W3SVC6 folder.

Moving to CloudFlare

Created 03 July 2018 07:11

I've been meaning to move to CloudFlare for a while, and thanks to this amazingly helpful page created by Troy Hunt, I've taken the leap.

I've been battling for time, and this really took all the research requirements right out of it, and I'm up and running with SSL even though my host doesn't support it on the cheap hosted package I'm using.

Changing CSS on MediaWiki

Created 26 June 2018 09:26, updated 26 June 2018 11:20

I've been using MediaWiki for ages for technical documentation, and I always forget how to implement custom CSS in both screen and print, so I'm adding this as a permanent reminder for myself.

To edit the CSS for the site (screen version), navigate to "http://YOURMEDIAWIKISITE/index.php?title=MediaWiki:Common.css" - and edit the CSS declared on that page

To edit the CSS for printing, navigate to "http://YOURMEDIAWIKISITE/index.php?title=MediaWiki:Print.css"

Easy!

Resolve Mixed Content Warnings with WordPress and the Enfold Theme

Created 03 May 2018 11:20

I needed to move a WordPress site to HTTPS. The certificate was installed, and the site was being correctly served over HTTPS, but I was getting mixed content warnings, as many of the images were being loaded over http. There are many sites that give the full rundown, but nothing worked for me for the Enfold theme.

This site: https://managewp.com/blog/wordpress-ssl-settings-and-how-to-resolve-mixed-content-warnings - can be used to set up HTTPS from start to finish

This site: https://isabelcastillo.com/mysql-wordpress-http-to-https - led me to the solution

Once the site was set up for https, I had to run the following four queries against the WordPress database:

UPDATE wp_posts 
SET    post_content = ( Replace (post_content, 'src="http://', 'src="//') )
WHERE  Instr(post_content, 'jpeg') > 0 
    OR Instr(post_content, 'jpg') > 0 
    OR Instr(post_content, 'gif') > 0 
    OR Instr(post_content, 'png') > 0;

UPDATE wp_posts
SET   post_content = ( Replace (post_content, "src='http://", "src='//") )
WHERE  Instr(post_content, 'jpeg') > 0 
    OR Instr(post_content, 'jpg') > 0 
    OR Instr(post_content, 'gif') > 0
    OR Instr(post_content, 'png') > 0;

UPDATE wp_postmeta
SET    meta_value = ( Replace (meta_value, 'src="http://', 'src="//') )
WHERE  Instr(meta_value, 'jpeg') > 0 
    OR Instr(meta_value, 'jpg') > 0 
    OR Instr(meta_value, 'gif') > 0 
    OR Instr(meta_value, 'png') > 0;

UPDATE wp_postmeta
SET    meta_value = ( Replace (meta_value, "src='http://", "src='//") )
WHERE  Instr(meta_value, 'jpeg') > 0 
    OR Instr(meta_value, 'jpg') > 0 
    OR Instr(meta_value, 'gif') > 0 
    OR Instr(meta_value, 'png') > 0;

Installing Windows 10 Pro over Home Edition

Created 26 February 2018 08:15, updated 26 February 2018 08:18

I purchased a new laptop through work today, which came with a pre-installed version of Windows Home Edition. I need Professional Edition for work, and so bought a copy of Windows Pro at the same time. Because I'm iffy about setup, I elected to install the new OS myself...what a mistake. No matter what I did, the Windows installer kept pulling the key from the BIOS, and so kept activating as Windows Home.

Things I tried that failed:

  1. Installing over Windows Home
  2. Deleting all partitions, re-partitioning, formatting all drives, and installing from scratch
  3. Swearing loudly, and reinstalling

Things I tried that worked:

  1. Download the Windows 10 ISO from Microsoft - click "Download tool now" and create a USB drive installation media (alternatively, you can download the ISO and install to the drive)
  2. I'm not 100% this step is necessary, but I did it anyway - create a "PID.txt" inside the "sources" folder file on your flash stick, with the following contents (where the XXXXX string is your valid Windows key)
    [PID]
    Value=XXXXX-XXXXX-XXXXX-XXXXX-XXXXX
    
  3. In the sources folder again, create a text file called ei.cfg and set the contents to as follows:
    [EditionID]
    Professional
    [Channel]
    OEM
    [VL]
    0
    
    In the below articles, it was mentioned that this file should be placed in x64sources and x86sources directories - for me it was just the "sources" folder
  4. Install Windows - I did this by just running the setup.exe file

I am documenting this here for my personal benefit, but this help as found here:

Importing files into SQL Server using OPENROWSET

Created 09 November 2017 12:17, updated 09 November 2017 12:45

I've used OPENROWSET to import Excel documents for years, but I was playing around today with CSV and Pipe-delimited files, and there are some tricks to these that I thought I would document for future reference.

All the examples below assume you have a folder F:\MyData on your server. I emphasise this, because if you run the below scripts using SSMS on your machine, that is not the same F: drive - it is the F: drive the server sees.

Excel

Excel documents go in really easily. You just need the name of the Sheet you want to import - in my example below I am importing all data from sheet "Sheet1" into table "MyExcelData".

SELECT * INTO MyExcelData FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0', 'Excel 12.0;Database=F:\MyData\MyFile.xlsx', [Sheet1$]) 

CSV

Second easiest, is a CSV document. Be warned, you will need to make sure that it is an ANSI-compliant file (or that there are no commas in the data), otherwise you will end up with columns named "F1", "F2", "F3" in your generated table.

select * INTO MyCsvData
FROM OPENROWSET('MSDASQL'
,'Driver={Microsoft Access Text Driver (*.txt, *.csv)};'''
,'SELECT * FROM F:\MyData\MyFile.csv')

Delimited text files (e.g. pipe-delimited)

Last, but not least, is a delimited document. In this case, it works similar to CSV, but when you try and import as per the above statement, you will end up with all the data in a single column. As such, you will need a Schema.ini file, that contains data about your file and is accessible to your server. In my example, I declare the DefaultDir as the place where the Schema.ini file resides.

Firstly, the Schema.ini. This file must contain details for each file that is being imported, where the heading of each section is the name of the file. So, if I am importing a pipe-delimited file called MyDelimitedFile.txt, that has headers in the column, my Schema.ini will have the following contents:

[MyDelimitedFile.txt]
    ColNameHeader=True
    Format=Delimited(|)

The SELECT statement must then reference the DefaultDir, and the delimiters will be used to determine the columns.

select * INTO MyDelimitedData
FROM OPENROWSET('MSDASQL'
,'Driver={Microsoft Access Text Driver (*.txt, *.csv)};DefaultDir=F:\MyData'''
,'SELECT * FROM F:\MyData\MyDelimitedFile.txt')

Load test with Fiddler

Created 20 September 2017 16:25

A client reported an issue with one of our API endpoints today with a rather depressing problem - it worked fine with single posts, but multiple concurrent posts resulted in random errors.

I had the means to test the API with Fiddler, but that didn't help me much as that only sends one request at a time and I couldn't replicate the issue. I have used West Wind Web Surge in the past for personal work, but it requires a license for commercial use and I couldn't get it to play nicely with a self-certified test ceritificate, so I started Googling for an alternative.

It turns out you can do it easily with Fiddler. Execute the request that you want to stress test once, and then select the request once the response is received. Hit Shift-R, and a pop-up will appear asking how many simultaneous requests you want to make, defaulting to 5. Click OK on that, and Fiddler will send 5 concurrent requests!

Web Site and Page Essentials

Created 24 August 2017 20:03, updated 02 July 2018 20:59
Creating a web site is simple, but there are a lot of basic items to consider that are easy to forget but will trip you up in the lifetime of the web site. These lists should be revisited regularly.

For your site as a whole

  1. Are JavaScript and CSS resources bundled and minimised?
  2. While sitemaps are not as important as they used to be, if you have a complex site heirarchy or some pages are perhaps hidden behind posted forms, it may help your SEO to include a sitemap (sitemap.xml)
  3. X-Frame-Options header
  4. If possible, serve everything over HTTPS (https://httpsiseasy.com/)
  5. Implement HSTS: https://www.hanselman.com/blog/HowToEnableHTTPStrictTransportSecurityHSTSInIIS7.aspx
  6. Ensure all passwords are hashed, and allow users to reset passwords by send them an email
  7. Error messages should be caught at a global level - don't reveal sensitive information when an error occurs
  8. Create an appropriate robots.txt
  9. Scan your site using http://www.webpagetest.org
  10. Scan your site using https://securityheaders.io/
  11. Run your site through https://asafaweb.com (particularly if your site is ASP.NET)
  12. Run your site through https://gtmetrix.com/
  13. Optimize your images: https://pnggauntlet.com/

For every page

  1. Version numbers on CSS and JS include files - this will prevent new deploys from going wrong because users have cached copies of old files on their machines e.g. Myfile.js?version=1.0.4
  2. SQL Injection attacks - have you parameterised all your queries? Dapper, maybe?
  3. CSRF/XSRF Vulnerabilities - every page on your site (apart from the login screen) should compare the value of a form item AND a cookie to ensure they are the same cyrptographically secure number, otherwise authentication fails. For ASP.NET users, use the anti-forgery token.
  4. Don't send any sensitive information over email e.g. passwords
  5. Can the parameters be tampered with to reveal data the user should not be able to get to?

Useful links

Extract SSH Key from Filezilla Client

Created 10 May 2017 15:52, updated 12 April 2019 10:40

I always forget this, so I'm documenting it here for myself. If you make SFTP connections from automated scripts or within a program, you really want to be supplying the SSH key.

On Windows, this can be extracted by doing the following:

  1. Open registry editor (regedit.exe)
  2. Go to HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys
  3. All the keys are stored, with types, on the right hand side under the SshHostKeys folder. The Name of each entry contains the host.
  4. Delete the key for the host you're looking for, and reconnect using Filezilla. It should supply you with a challenge response with the key and value for future use. In code, you will use a combination of the Hostkey algorithm and the fingerprint. So, for this response from FileZilla:

    In code this would be referenced as:
    SshHostKeyFingerprint = "ecdsa-sha2-nistp384 384 b1:88:d6:eb:83:18:7e:80:77:9f:ef:xx:xx:xx:xx:xx"
    
  5. You may want to confirm this key with the owner of the source system!