Category: Technology

Oracle – Adding Fixed String To Column Data

I had a request to drop data from a SQL query into an Excel spreadsheet — the initial request had a column that included different text depending on row data. Since Box::Spout deals well with huge volumes of data (I write millions of rows in some reports), I use it pretty exclusively. Writing data to each column individually means I’ve got to retain a list of column names, and I would rather not do that. I can dump the column names into a header row then dump the entire database row into the next spreadsheet row. This works since a 1:1 correlation between database rows and Excel rows. I’ll go through after the fact and update a specific cell based on data in other cells, but I don’t want to insert a column.

Luckily, there’s an easy way to add a placeholder column to my database output:

SELECT DISTINCT MAX(DLR1.ISSUE_NBR) MAX_ISSUE_NBR, DLR.ISSUE_NBR, ‘MatchIndicator’ as “MatchIndicator”, DLR.EQUIPMENT_ID, EQ.EQUIPMENT_ID, C.CIRCUIT_DESIGN_ID, C.EXCHANGE_CARRIER_CIRCUIT_ID, C.STATUS, C.RATE_CODE, C.SERVICE_TYPE_CATEGORY, C.SERVICE_TYPE_CODE, NL.LOCATION_ID, NL.CLLI_CODE, DLR.LOCATION, DLR.BLOCK_IND

Filtering HTML Drop-down

I’ve got a few drop-downs that I’ve added filtering on the drop-down – start typing and you’ll see the options that match your string. But I needed to mirror an application functionality where you select a category and are then presented with a list of options that fit the category.

Here’s the drop-down selector for the categories

    echo "      <div class=\"row\">\n";
    echo "          <div class=\"col-md-12 col-sm-12 col-lg-12 col-xs-12\">\n";
    echo "              <div class=\"row\">\n";
    echo "                  <div class=\"row\">\n";
    echo "                      <div class=\"col-md-2 col-sm-2 col-lg-2 col-xs-2 text-left\">\n";
    echo "                          <span><strong>Animal Category:</strong></span>\n";
    echo "                      </div>\n";
    echo "                      <div class=\"col-md-10 col-sm-10 col-lg-10 col-xs-10 text-left form-group\">\n";
    echo "                          <select name=\"strAnimalType\" id=\"strAnimalType\" readonly/> \n";
    echo "                              <option class=\"NoSelection\" value=\"-----\">-----</option>\n";
    echo "                              <option class=\"Feline\" value=\"201\">Feline</option>\n";
    echo "                              <option class=\"Canine\" value=\"202\">Canine</option>\n";
    echo "                              <option class=\"Equine\" value=\"203\">Equine</option>\n";
    echo "                              <option class=\"Other\" value=\"204\">Other</option>\n";
    echo "                          </select>\n";
    echo "                      </div>\n";
    echo "                  </div>\n";

And here’s the drop-down selector I want to filter based on category — there are a lot of options. The class for each option includes the category selectors that will include the option in the drop-down.

    echo "      <div class=\"row\">\n";
    echo "          <div class=\"col-md-12 col-sm-12 col-lg-12 col-xs-12\">\n";
    echo "              <div class=\"row\">\n";
    echo "                  <div class=\"row\">\n";
    echo "                      <div class=\"col-md-2 col-sm-2 col-lg-2 col-xs-2 text-left\">\n";
    echo "                          <span><strong>Pet Breed:</strong></span>\n";
    echo "                      </div>\n";
    echo "                      <div class=\"col-md-10 col-sm-10 col-lg-10 col-xs-10 text-left form-group\">\n";
    echo "                          <select name=\"strPetBreed\" id=\"strPetBreed\" readonly/> \n";
    echo " <option value=\"-----\" class=\"selectors All\">-----</option>\n";
    echo " <option value=\"101\" class=\"selectors Feline\">Domestic Shorthair</option>\n";
    echo " <option value=\"1275\" class=\"selectors Feline\">Flame Point Siamese</option>\n";
    echo " <option value=\"1069\" class=\"selectors Equine\">Arabian</option>\n";
    echo " <option value=\"1071\" class=\"selectors Equine\">Tennessee Walking Horse</option>\n";
    echo " <option value=\"1072\" class=\"selectors Other\">Chicken</option>\n";
    echo " <option value=\"1073\" class=\"selectors Other\">Snake</option>\n";
    echo " <option value=\"1074\" class=\"selectors Canine\">Australian Shepherd</option>\n";
    echo " <option value=\"1075\" class=\"selectors Feline\">Burmese</option>\n";
    echo " <option value=\"1076\" class=\"selectors Canine\">Siberian Husky</option>\n";
    echo " <option value=\"1077\" class=\"selectors Feline\">Sphinx</option>\n";
    echo " <option value=\"1078\" class=\"selectors Other\">Rabbit</option>\n";
    echo "                          </select>\n";
    echo "                      </div>\n";
    echo "                  </div>\n";
    echo "              </div>\n";
    echo "          </div>\n";
    echo "      </div>\n";

In the JavaScript, I’ve got a quick function that repopulates the rather long drop-down menu based on the selected category

// Filter strPetBreed options based on strAnimalCategory value
$(document).ready(function () {    
    var allOptions = $('#strPetBreedoption')
    $('#strAnimalCategory').change(function () {
        $('#strPetBreed option').remove()

        var classN = $('#strAnimalCategory option:selected').prop('class');
        var optsCat = allOptions.filter('.' + classN);
        $.each(optsCat, function (i, j) {
            $(j).appendTo('#strPetBreed');
        });

        var optsAll = allOptions.filter('.All');
        $.each(optsAll, function (i, j) {
            $(j).prependTo('#strPetBreed');
        });

    });
});

Since it’s possible there are options you’d want to always appear (in my case, it’s just the “—–” to indicate no selection has been made … but there could be real items that fall into each category too), I’ve got an “All” classification that will get popped onto the top of the list.

NodeJS Unit File

For future reference, this is an example unit file for running a NodeJS server with systemd. The NodeJS code we use reads from a local MariaDB, so I’ve added a service dependency for the database server.

Create /etc/systemd/system/nodeserver.service

[Unit]
Description=SiteName Node.js Server
Requires=After=mariadb.service

[Service]
ExecStart=/path/to/binary/for/node /path/to/nodeJS/html/server.js
WorkingDirectory=/path/to/nodeJS/html
Restart=always
RestartSec=30
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=nodejs-sitename
User=web-user
Group=www-group

[Install]
WantedBy=multi-user.target

Use systemctl daemon-reload to register the new unit file, then “systemctl start nodeserver.service” to start the service. Assuming everything works properly, use “systemctl enable nodeserver.service” to have the service start on boot.

LDAP Authentication: Python Flask

This is a quick python script showing how the flask-ldap3-login module can be used to authenticate and gather user attribute values

from flask_ldap3_login import LDAP3LoginManager
from ldap3 import Tls
import ssl

config = dict()

config['LDAP_HOST'] = 'ad.example.com'

# Use SSL unless you are debugging a problem. Clear text port is 389 and tls_ctx needs to be removed from add_server call
config['LDAP_USE_SSL'] = True
config['LDAP_PORT'] = 636

# Base DN
config['LDAP_BASE_DN'] = 'dc=example,dc=com'

# User Base DN, prepended to Base DN
config['LDAP_USER_DN'] = 'ou=UserDN'

# Groups Base DN, prepended to Base DN
config['LDAP_GROUP_DN'] = 'ou=SecurityGroupDN'

# Server will be manually added to establish SSL
config['LDAP_ADD_SERVER'] = False

# Domain component of userprincipal name
config['LDAP_BIND_DIRECT_SUFFIX'] = '@example.com'

# Search scope needs to be subtree
config['LDAP_USER_SEARCH_SCOPE'] = "SUBTREE"

# Attributes to return
config['LDAP_GET_USER_ATTRIBUTES'] = ("mail", "givenName", "sn")

# Setup a LDAP3 Login Manager.
ldap_manager = LDAP3LoginManager()

# Init the mamager with the config since we aren't using an app
ldap_manager.init_config(config)

# TLS settings to establish trust without validating CA issuance chain. 
# Can use CERT_REQUIRED and ca_certs_file with path to cacerts that includes issuing chain
tls_ctx = Tls(
    validate=ssl.CERT_NONE,
    version=ssl.PROTOCOL_TLSv1,
    valid_names=[
        'ad.example.com',
    ]
)

ldap_manager.add_server(
    config.get('LDAP_HOST'),
    config.get('LDAP_PORT'),
    config.get('LDAP_USE_SSL'),
    tls_ctx=tls_ctx
)

# Validate credentials
response = ldap_manager.authenticate_direct_credentials('e0012345', 'P@s5w0rdG03sH3re')
print(response.status)
print(response.user_info)

LDAP Authentication and Authorization: PHP

Blah

<?php
    error_reporting(0);
    #=== FUNCTION ==================================================================
    #      NAME: ldapAuthenticationAndAuthorizationWithAttributes
    #      PARAMETERS:
    #                    $strLDAPHost                   String  LDAP Server URI
    #                    $strUIDAttr                    String  Schema attribute for user ID search
    #                    $strSystemUser                 String  System credential username
    #                    $strSystemPassword             String  System credential password
    #                    $strUserBaseDN                 String  User search LDAP base DN
    #                    $strLogonUserID                String  Input user ID
    #                    $strLogonUserPassword          String  Input user password
    #					 $arrayAttrsToReturn			String	Attributes to be returned
    #                    $strGroupBaseDN                String  (optional) Group search LDAP base DN
    #                    $strGroupNamingAttribute       String  (optional) Schema attribute for group search
    #                    $strMembershipAttr             String  (optional) Schema attribute for group membership
    #                    $strAuthGroup                  String  (optional) Group name
    #     DESCRIPTION: Verify authentication and authorization against AD server.a
    #
    #     RETURNS: array(BindReturnCode, Authorized, array(returnValues))
    #                        BindReturnCode:    -1 indicates LDAP connection failure, -2 indicates system account auth failed, -3 indicates user auth not attempted, >=0 is IANA-registered resultCode values (https://www.iana.org/assignments/ldap-parameters/ldap-parameters.xml#ldap-parameters-6)
    #							NOTE: 0 is successful authentication in IANA-registered resultCode
    #                        Authorized:        0 authorization not attempted, -1 is not a member of the located group, 1 is member of the located group
    #						arrayUserAttributeValues	Array with values of $arrayAttrsToReturn
    #
    #     USAGE: $arrayUserAuthorized = ldapAuthenticationAndAuthorizationWithAttributes("ldaps://ad.example.com","sAMAccountName","ldapquery@example.com", "Sy5t3mP@ssw0rdG03sH3re", "ou=example,dc=example,dc=com", $strInputUserName, $strInputUserPassword, array('givenName', 'sn'), "ou=securitygroups,dc=example,dc=com","cn", "member", "LJRTestGroup")
    #===============================================================================
    function ldapAuthenticationAndAuthorizationWithAttributes($strLDAPHost,$strUIDAttr, $strSystemUser, $strSystemPassword, $strUserBaseDN, $strLogonUserID, $strLogonUserPassword, $arrayAttrsToReturn, $strGroupBaseDN=null, $strGroupNamingAttribute=null, $strMembershipAttr=null, $strAuthGroup=null){
        $arrayAuthResults = array();
        $arrayUserAttributeValues = array();
        // Validate password is not null, otherwise directory servers implementing unauthenticated bind (https://tools.ietf.org/html/rfc4513#section-5.1.2) will return 0 on auth attempts with null password
        if( strlen($strLogonUserPassword) < 1){
            $arrayAuthResults['BindReturnCode'] = -3;
            $arrayAuthResults['Authorized'] = -1;
        }
        else{
            // Connect to the LDAP directory for system ID queries
            $systemDS = ldap_connect($strLDAPHost);
            ldap_set_option($systemDS, LDAP_OPT_PROTOCOL_VERSION, 3);

            if ($systemDS) {
                // Bind with the system ID and find $strLogonUserID FQDN
                $systemBind = ldap_bind($systemDS, $strSystemUser, $strSystemPassword);

                if(ldap_errno($systemDS) == 0){
                    $strLDAPFilter="(&($strUIDAttr=$strLogonUserID))";
                    $result=ldap_search($systemDS,$strUserBaseDN,$strLDAPFilter, $arrayAttrsToReturn);

                    $entry = ldap_first_entry($systemDS, $result);

                    $strFoundUserFQDN= ldap_get_dn($systemDS, $entry);

                    if($strFoundUserFQDN){
                        $userDS = ldap_connect($strLDAPHost);
                        ldap_set_option($userDS, LDAP_OPT_PROTOCOL_VERSION, 3);

                        $userBind = ldap_bind($userDS, $strFoundUserFQDN, $strLogonUserPassword);
                        $arrayAuthResults['BindReturnCode'] = ldap_errno($userDS);

                        ldap_close($userDS);

                        if($arrayAuthResults['BindReturnCode'] == 0){
                        	$objFoundUser = ldap_get_entries($systemDS, $result);
							for($arrayAttrsToReturn as $strAttributeName){
								$arrayUserAttributeValues[$strAttributeName] = $objFoundUser[0][$strAttributeName];

							}
							$arrayAuthResults['AttributeValues'] = $arrayUserAttributeValues;
                            //////////////////////////////////////////////////////////////////////////////////////
                            // If an auth group has been supplied, verify authorization
                            //////////////////////////////////////////////////////////////////////////////////////
                            if($strAuthGroup){
								// Escapes in DN need to be double-escaped or bad search filter error is encountered
                                $strGroupQuery = "(&($strGroupNamingAttribute=$strAuthGroup)($strMembershipAttr=" . str_replace("\\","\\\\", $strFoundUserFQDN) . "))";

                                $groupResult = ldap_search($systemDS,$strGroupBaseDN, $strGroupQuery);
                                $authorisedState = ldap_count_entries($systemDS ,$groupResult);

                                // If a group matching the filter is found, the user is authorised
                                if($authorisedState == 1){
                                    $arrayAuthResults['Authorized'] = 1;
                                }
                                // Otherwise the user is not a member of the group and is not authorised
                                else{
                                    $arrayAuthResults['Authorized'] = -1;
                                }
                            }
                            else{
                                $arrayAuthResults['Authorized'] = 0;
                            }
                            //////////////////////////////////////////////////////////////////////////////////////
                            ldap_close($systemDS);
                        }
                        // If the bind failed, the user has not logged in successfully so they cannot be authorized
                        else{
                            $arrayAuthResults['Authorized'] = -1;

                            ldap_close($systemDS);
                            ldap_close($userDS);
                        }
                    }
                    // User not found in directory
                    else{
                        $arrayAuthResults['BindReturnCode'] = 32;
                        $arrayAuthResults['Authorized'] = -1;
                    }
                }
                // system bind failed
                else{
                    $arrayAuthResults['BindReturnCode'] = -2;
                    $arrayAuthResults['Authorized'] = -1;
                    ldap_close($systemDS);
                }
            }
            // ldap connection failed
            else{
                $arrayAuthResults['BindReturnCode'] = -1;
                $arrayAuthResults['Authorized'] = -1;
            }
        }
        return $arrayAuthResults;
    }

    print "User password not supplied:\n";
    $arrayNullPassword = array();
    $arrayNullPassword = ldapAuthenticationAndAuthorizationWithAttributes("ldaps://ad.example.com","sAMAccountName","ldapquery@example.com", "Sy5t3mP@ssw0rdG03sH3re", "ou=example,dc=example,dc=com", "e0012345", '');
    var_dump($arrayNullPassword);

    print "Bad password:\n";
    $arrayBadPassword = array();
    $arrayBadPassword = ldapAuthenticationAndAuthorizationWithAttributes("ldaps://ad.example.com","sAMAccountName","ldapquery@example.com", "Sy5t3mP@ssw0rdG03sH3re", "ou=example,dc=example,dc=com", "e0012345", 'N0tTh3P@s5w0rd',"ou=SecurityGroups,dc=example,dc=com","cn", "member");
    var_dump($arrayBadPassword);

    print "\nInvalid user:\n";
    $arrayUserNotInDirectory = array();
    $arrayUserNotInDirectory = ldapAuthenticationAndAuthorizationWithAttributes("ldaps://ad.example.com","sAMAccountName","ldapquery@example.com", "Sy5t3mP@ssw0rdG03sH3re", "ou=example,dc=example,dc=com", "xe0012345", 'xDoesN0tM@tt3r');
    var_dump($arrayUserNotInDirectory);

    print "\nGood password without authorization:\n";
    $arrayUserAuthenticated = array();
    $arrayUserAuthenticated = ldapAuthenticationAndAuthorizationWithAttributes("ldaps://ad.example.com","sAMAccountName","ldapquery@example.com", "Sy5t3mP@ssw0rdG03sH3re", "ou=example,dc=example,dc=com", "e0012345", 'Us3rP@s5w0rdG035H3re|Us3rP@s5w0rdG035H3re');
    var_dump($arrayUserAuthenticated);

    print "\nGood password with authorized user:\n";
    $arrayUserAuthorized = array();
    $arrayUserAuthorized = ldapAuthenticationAndAuthorizationWithAttributes("ldaps://ad.example.com","sAMAccountName","ldapquery@example.com", "Sy5t3mP@ssw0rdG03sH3re", "ou=example,dc=example,dc=com", "e0012345", 'Us3rP@s5w0rdG035H3re|Us3rP@s5w0rdG035H3re',"ou=SecurityGroups,dc=example,dc=com","cn", "member", "cfyP_Unix_UnixUsers");
    var_dump($arrayUserAuthorized);

    print "\nGood password with unauthorized user:\n";
    $arrayUserNotAuthorized = array();
    $arrayUserNotAuthorized = ldapAuthenticationAndAuthorizationWithAttributes("ldaps://ad.example.com","sAMAccountName","ldapquery@example.com", "Sy5t3mP@ssw0rdG03sH3re", "ou=example,dc=example,dc=com", "e0012345", 'Us3rP@s5w0rdG035H3re|Us3rP@s5w0rdG035H3re',"ou=SecurityGroups,dc=example,dc=com","cn", "member", "WIN AM Team West");
    var_dump($arrayUserNotAuthorized);

    print "\nBad system account:\n";
    $arrayBadSystemCred = array();
    $arrayBadSystemCred = ldapAuthenticationAndAuthorizationWithAttributes("ldaps://ad.example.com","sAMAccountName","ldapquery@example.com", "xSy5t3mP@ssw0rdG03sH3re", "ou=example,dc=example,dc=com", "e0012345", 'Us3rP@s5w0rdG035H3re|Us3rP@s5w0rdG035H3re');
    var_dump($arrayBadSystemCred);

?>

LDAP Authentication: PHP and Active Directory

This is a very brief function that authenticates a user against Active Directory. Because you can authenticate using a fully qualified DN, sAMAccountName, or userPrincipalName … there’s no need to use a system credential or search for the user provided you’ve got a single domain in your forest (i.e. you know what to prepend to the sAMAccountName or postpend to userPrincipalName).

If you need to perform authorization as well as authentication, you’ll need the user’s FQDN so use the generic LDAP authentication and authorization function.

<?php
    error_reporting(0);
    #=== FUNCTION ==================================================================
    #      NAME: activeDirectoryLDAPAuthentication
    #      PARAMETERS: 
    #                    $strLDAPHost                   String  LDAP Server URI
    #                    $strLogonUserID                String  Input user ID
    #                    $strLogonUserPassword          String  Input user password
    #     DESCRIPTION: Verify authentication againt Active Directory server.
    #     
    #     RETURNS: int BindReturnCode:    -2 indicates LDAP connection failure, -3 indicates user auth not attempted, >=0 is IANA-registered resultCode values (https://www.iana.org/assignments/ldap-parameters/ldap-parameters.xml#ldap-parameters-6)
    #							NOTE: 0 is successful authentication in IANA-registered resultCode
    #
    #     USAGE: $iBindResult = activeDirectoryLDAPAuthentication("ldaps://ad.example.com", $strInputUserName, $strInputUserPassword)
    #===============================================================================
    function activeDirectoryLDAPAuthentication($strLDAPHost, $strLogonUserID, $strLogonUserPassword){
        $iBindReturnCode = null;
        // Validate password is not null, otherwise directory servers implementing unauthenticated bind (https://tools.ietf.org/html/rfc4513#section-5.1.2) will return 0 on auth attempts with null password
        if( strlen($strLogonUserPassword) < 1){
            $iBindReturnCode = -1;
        }
        else{
            $userDS = ldap_connect($strLDAPHost);
            if($userDS){
                ldap_set_option($userDS, LDAP_OPT_PROTOCOL_VERSION, 3);

                $userBind = ldap_bind($userDS, $strLogonUserID . '@example.com', $strLogonUserPassword);
                $iBindReturnCode = ldap_errno($userDS);
                ldap_close($userDS);
            }
            // ldap connection failed
            else{
                $iBindReturnCode = -2;              
            }        
        }
        return $iBindReturnCode;
    }

    $iBadUser = activeDirectoryLDAPAuthentication("ldaps://ad.example.com", "xe0012345", 'N0tTh3P@s5w0rd');
    print "\nInvalid user: $iBadUser\n";

    $iUserAuthenticated = activeDirectoryLDAPAuthentication("ldaps://ad.example.com", "e012345", 'Go0dP@s5w0rdH3r3');
    print "\nGood password: $iUserAuthenticated\n";

    $iBadPassword = activeDirectoryLDAPAuthentication("ldaps://ad.example.com", "e0012345", 'N0tTh3P@s5w0rd');
    print "\nBad password: $iBadPassword\n";

    $iBadHost = activeDirectoryLDAPAuthentication("ldaps://abc.example.com", "e0012345", 'N0tTh3P@s5w0rd');
    print "\nBad host: $iBadHost\n";

?>


Oracle Password Expiry – Sandbox Server

Oracle 11g seems to ship with password expiry enabled — which is a very good thing for production systems. I’ve even written some code to maintain our system account password (scripts are grabbing the password from a not-clear-text storage facility anyway, so it wasn’t a big deal to add an n-1 password and move the current stashed password into the n-1 column, change the account password, and stash the updated password in the current password location … now my system ID password is updated by a monthly cron job, no one actually knows the password {although anyone could find it, so I would run the password cycle script when individuals leave the group}). But I’m a lot lazier about this stuff in my sandbox. Proof of concept code has clear text passwords. But the server is bound to localhost & there’s no real data in, well, anything.

I started seeing lines in my error log indicating the password would expire. Aaaand that’s how I learned that password expiry was enabled by default now.

[Sat Apr 18 07:42:59 2020] [error] [client 127.0.0.1] PHP Warning: oci_connect(): OCI_SUCCESS_WITH_INFO: ORA-28002: the password will expire within 7 days in /var/www/vhtml/…/file.php on line 191, referer: …

I’m going to disable password expiry because it’s a sandbox. For a real system, obviously, this may not be a stellar idea.

select USERNAME, ACCOUNT_STATUS, PROFILE from dba_users where USERNAME = 'SampleUser';

 

USERNAME ACCOUNT_STATUS PROFILE
SampleUser EXPIRED(GRACE) DEFAULT

 

Note the account status “EXPIRED(GRACE)” — that’s why I am getting the error shown above. Grab the profile name — it’s a sandbox, so 99% sure it’s going to be ‘DEFAULT’ and alter that profile with an unlimited password expiration:

alter profile <profile_name> limit password_life_time UNLIMITED;

Except that didn’t actually stop the error. Turns out you’ve still got to change the password once the account has been flagged as expired (or let the password expire and then unlock the account … but I was looking at the log because I’m debugging something, and I wanted the error to stop *right now*).

alter user SampleUser identified by N3W_P@s5_w0rD;

 

Did you know … you can list the members of your MS Team?

Without any special Administrative rights, you can list the members of the Azure AD groups that are used in MS Teams. If you don’t already have the AzureAD module installed, install it. In Windows, this is:

Install-Module -Name AzureAD

In Linux,you’ll need the preview of Azure AD:

# To run on Linux, you need the preview mode of AzureAD
Register-PackageSource -Trusted -ProviderName 'PowerShellGet' -Name 'Posh Test Gallery' -Location https://www.poshtestgallery.com/api/v2/
Install-Module -Name AzureAD.Standard.Preview
 

Connect to AzureAD. There is a separate command to list the group owners (Get-AzureADGroupOwner). I’ve always found the owner(s) in the member list as well, but it’s technically possible to have entries unique to the owner list.

Connect-AzureAD
Get-AzureADGroup -SearchString "Group Name Here" | Get-AzureADGroupOwner -All $True
Get-AzureADGroup -SearchString "Group Name Here" | Get-AzureADGroupMember -All $True

Redirect the output to a file if you wish to use the results elsewhere, or stash the returned member list in a variable and use Get-AzureADUser to get additional information for the user records.

$objMembers | ForEach-Object -Process {get-azureaduser -objectid $_.ObjectID}

Open Source Methodologies – Design Document Driven

We use design documents at work to ensure a clear understanding between the requestors, product owners, and developers. There’s a request document that outlines what they’re looking to accomplish, we produce a design document that outlines what we’re doing and how that will be accomplished. External dependencies use the design document to implement their required services — if I have a design document that says I’ll pass x (required), y (required), and z (optional) to a WSDL and end up with an object in the application database where a=x, b=y, and, optionally, c=z … they’ll whip up an endpoint that takes the parameters, performs the required actions, and builds the object I need. Once everyone is in agreement that it’s what they want, it’s reasonable (security, ROI), and possible … developers get to work. Tests are built against the documented functionality, and we know we’re done when the tests pass. If the users want something changed, the design document is amended, a quick feasibility/reason-ability review is performed, and development work commences.

I thought of this process after observing some people push PRs for major overhaults into a few open source projects only to have the PRs rejected as, essentially, not the direction “we” want to go. On the other extreme … I’ve made some changes to open source apps — in some cases, those were bug fixes, and I’ve pushed the changes back to the main project. But, when I’ve changed functionality. I’ve made those changes to address a specific need I have, and I leave the changes in my own fork. Which has the detriment of, potentially, not providing useful functionality to the main project. While it’s perfectly reasonable to put a lot of time into a major change that you need anyway (and, potentially, offer those changes back to the community) … it is terrible to put a lot of your time into something for someone else and have it rejected. And while not engaging with the project maintainers to see if they’re interested in my derivative work saves effort for me, it reduces innovation (how many people actually run through all of the forks of a project to see if anything ‘interesting’ happened elsewhere?).

Obviously, the answer isn’t for projects to accept effort just because it’s a significant investment on the contributor’s part — there should be some mechanism for ensuring what you’re about to delve into is something the project maintainers actually want. Which is what made me think of the design documents we use at work.

If a project had a design document that detailed what it waned to do, how it was doing it, and potentially a section for desired future features and functionality … it would provide a guideline to anyone looking to contribute. A change that doesn’t impact functionality (e.g. a bugfix) can be worked on and submitted for inclusion in the project as occurs normally. If you want to change something about how the application works, you first submit a PR against the design document. Outline what you want to do, how you want to do it … maybe even a quick mock-up (I use Pencil Project for mock-ups where I’m uncertain the final project will be approved, HTML web code when I know the project is a go and want a head-start on my development … but I also accept the risk that I ‘wasted’ a few hours building the design document wireframe if the project gets dropped). Want to work on one of the ‘desired future’ items? Modify the design document to include that functionality and how it’ll be implemented. Discussion about the approach refines what you’ll actually be doing, and you’ll understand if the project maintainers are interested in your contribution before dedicating hundreds of hours to development.

Some projects are interested in “bugfixes only” — which can be stated in the contributing guidelines. In this example, I developed a quick script to produce in-scope user lists. I don’t want to include other details about the identified users, I don’t want to find the first n levels of reports, etc. It’s exactly what I needed, and I’m putting it on GitHub as an example — using Python to search an LDAP directory and using recursion with back-linked attributes.

Projects that are open for collaborative contributions, though, can include the design document location in their contributing guidelines. Initial contributions are made against the design document, discussed, and approved or rejected. Once approved, code can be developed to the new design.