Friday, August 25, 2017

Windows PE and Cisco ISE authentication 802.1x

Windows PE  and Cisco ISE authentication

This blog entry is intended to assist you when implementing a Cisco ISE next generation network across the organisation.  Without ISE profiles the SCCM Task Sequence will fail to connect to Distribution Points and the MDT database.  UNC paths are blocked and network access is restricted.
Cisco ISE by design will restrict network access to prevent unauthorized clients from simply plugging their equipment into the network and being routed like a authorised client.
Computer and User Authentication (explain in detailed section)
Cisco ISE profiles should be implemented in two ways; Cisco ISE profiles via Group Policy for domain joined systems, and to bake ISE profiles into the SCCM Boot Image.  The guide below will explain how to implement both configuration setups.


Tutorial - WinPE

Microsoft has detailed the two XML files required to achieve User Authentication when in WinPE here

Create an XML called "EthernetLANProfile.xml" containing the following. The Thumb Print detailed within <TrustedRootCA> should reflect a trusted Third Party Cert; This ISE certification should also be deployed to all Domain Joined systems for GPO ISE Profiles (see below Tutorial - OS)

<?xml version="1.0"?>
<!-- Sample LAN profile: EthernetLANProfile.xml" -->
<LANProfile xmlns="http://www.microsoft.com/networking/LAN/profile/v1">
  <MSM>
    <security>
      <OneXEnforced>false</OneXEnforced>
      <OneXEnabled>true</OneXEnabled>
      <OneX xmlns="http://www.microsoft.com/networking/OneX/v1">
        <cacheUserData>true</cacheUserData>
        <authMode>user</authMode>
        <EAPConfig><EapHostConfig
          xmlns="http://www.microsoft.com/provisioning/EapHostConfig"><EapMethod><Type
          xmlns="http://www.microsoft.com/provisioning/EapCommon">25</Type><VendorId
          xmlns="http://www.microsoft.com/provisioning/EapCommon">0</VendorId><VendorType
          xmlns="http://www.microsoft.com/provisioning/EapCommon">0</VendorType><AuthorId
          xmlns="http://www.microsoft.com/provisioning/EapCommon">0</AuthorId></EapMethod><Config
          xmlns="http://www.microsoft.com/provisioning/EapHostConfig"><Eap
          xmlns="http://www.microsoft.com/provisioning/BaseEapConnectionPropertiesV1">
        <Type>25</Type><EapType
          xmlns="http://www.microsoft.com/provisioning/MsPeapConnectionPropertiesV1">
        <ServerValidation>
          <DisableUserPromptForServerValidation>false</DisableUserPromptForServerValidation>
          <ServerNames></ServerNames>
          <TrustedRootCA>1a 2b 3c 4d 56 78 90 aa bb cc dd ee ff 1a 2b 3c 4d 5e 6f</TrustedRootCA>
          </ServerValidation><FastReconnect>true</FastReconnect>
          <InnerEapOptional>false</InnerEapOptional><Eap
            xmlns="http://www.microsoft.com/provisioning/BaseEapConnectionPropertiesV1">
          <Type>26</Type><EapType
            xmlns="http://www.microsoft.com/provisioning/MsChapV2ConnectionPropertiesV1">
          <UseWinLogonCredentials>false</UseWinLogonCredentials></EapType></Eap>
          <EnableQuarantineChecks>false</EnableQuarantineChecks>
          <RequireCryptoBinding>false</RequireCryptoBinding><PeapExtensions>
          <PerformServerValidation
            xmlns="http://www.microsoft.com/provisioning/MsPeapConnectionPropertiesV2">false
          </PerformServerValidation><AcceptServerName
            xmlns="http://www.microsoft.com/provisioning/MsPeapConnectionPropertiesV2">false
            </AcceptServerName><PeapExtensionsV2
            xmlns="http://www.microsoft.com/provisioning/MsPeapConnectionPropertiesV2">
          <AllowPromptingWhenServerCANotFound
            xmlns="http://www.microsoft.com/provisioning/MsPeapConnectionPropertiesV3">true
          </AllowPromptingWhenServerCANotFound></PeapExtensionsV2></PeapExtensions></EapType>
        </Eap></Config></EapHostConfig></EAPConfig>
      </OneX>
    </security>
  </MSM>
</LANProfile>



Create another XML file called "EAP_UserData.xml" containing the Service Account User Credentials.

<?xml version="1.0"?> <!-- Sample EAP user data: EAP_UserData.xml" --> <EapHostUserCredentials xmlns="http://www.microsoft.com/provisioning/EapHostUserCredentials" xmlns:eapCommon="http://www.microsoft.com/provisioning/EapCommon" xmlns:baseEap="http://www.microsoft.com/provisioning/BaseEapMethodUserCredentials"> <EapMethod> <eapCommon:Type>25</eapCommon:Type> <eapCommon:AuthorId>0</eapCommon:AuthorId> </EapMethod> <Credentials xmlns:eapUser="http://www.microsoft.com/provisioning/EapUserPropertiesV1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:baseEap="http://www.microsoft.com/provisioning/BaseEapUserPropertiesV1" xmlns:MsPeap="http://www.microsoft.com/provisioning/MsPeapUserPropertiesV1" xmlns:MsChapV2="http://www.microsoft.com/provisioning/MsChapV2UserPropertiesV1"> <baseEap:Eap> <baseEap:Type>25</baseEap:Type> <MsPeap:EapType> <MsPeap:RoutingIdentity>onex\administrator</MsPeap:RoutingIdentity> <baseEap:Eap> <baseEap:Type>26</baseEap:Type> <MsChapV2:EapType> <MsChapV2:Username>SVC-Account-Name</MsChapV2:Username> <MsChapV2:Password>SVC-Account-password</MsChapV2:Password>
<MsChapV2:LogonDomain>SVC-Account-Domain</MsChapV2:LogonDomain>
</MsChapV2:EapType> </baseEap:Eap> </MsPeap:EapType> </baseEap:Eap> </Credentials> </EapHostUserCredentials>


The SCCM Boot image will need to contain the WinPE-Dot3Svc Optional Component.  However, from my experience this component doesnt work in Windows 10 version 1607 or 1703.
The component wizard completes without errors however, the Dot3Svc cannot start in WinPE. Micrsoft have detailed the issue here

The component will need to be manually installed; download the KB4025632 MSU file from Windows Update Catalog- ## https://www.catalog.update.microsoft.com/Search.aspx?q=KB4025632

The following two command will mount the Boot.wim Image and inject the Dot3Svc component.

Dism /Mount-Image /ImageFile:"C:\temp\WinPEx64\sources\boot.wim" /index:1 /MountDir:"C:\temp\WinPE_amd64-mount"

Dism /Add-Package /Image:"C:\temp\WinPE_amd64-mount" /PackagePath:"C:\temp\WinPEx64\windows10.0-kb4025632-x64_af86717e4eec306948b23cd1e82ff95640e51f5e.msu"

Before the Boot image is dismounted and copied to SCCM we also need to bake the ISE XML profiles into the Boot image.

Copy the EthernetLANProfile.xml & EAP_UserData.xml created earlier into the folder  "windows\system32"

C:\temp\WinPE_amd64-mount\windows\system32\

Dism /Unmount-Image /MountDir:"C:\temp\WinPE_amd64-mount" /commit

After installing the component copy the wim to your Boot image source location.
Add a custom prestart command.  Open the Properties of the Boot image and go to the customization tab, enable the prestart command and type the following three commands (enable Dot3svc, import Ethernet Profile , import User Auth Profile"

cmd /c powershell -noninteractive -command net start dot3svc & cmd /c netsh lan add profile filename=%SYSTEMDRIVE%\windows\system32\EthernetLANProfile.xml interface=*  & cmd /c netsh lan set eapuserdata filename="%SYSTEMDRIVE%\windows\system32\EAP_UserData.xml" alluser=yes Interface=*


 Within the SCCM console update the distribution point to inject the SCCM binaries and distribute the WIM to your PXE enabled distribution Points.

Once the Boot image is loaded and you have typed your WinPE password (if present) the Prestart command will launch (Custom Hook).  WinPE will run the commands in the TSConfig.ini file located on the root of the X drive.

Before the list of Task Sequences (if available) are presented you will see a command window appear starting the dot3svc service and configure the User Authentication ISE profiles created earlier.

If you wanted to check that ISE is running before kicking off the Task Sequence then you can:-

Hit F8 for a command prompt (if enabled in the boot image)
Type:
 PowerShell -command Get-Service dot3svc 

You should see the service status as running

Running       dot3svc            Wired AutoConfig





    Monday, August 14, 2017

    ScanAgent.log reports OnScanComplete with error=0x87d00692

    ScanAgent.log reports OnScanComplete with error=0x87d00692


    Scenario:

    Clients are not receiving Software Updates for both Microsoft OS updates and Windows Defender.  The SCCM concole resports that the system does not summaries any required updates. We have a Group policy in place applying a specific update service location; this location is applied in several policies over several OU's. Only one policy when applied results in the error 0x87d00692.

    ScanAgent.log

    ScanJob({BAA53BE2-9FE1-48B6-89A7-0D60BC07ED50}): CScanJob::OnScanComplete -Scan Failed with Error=0x87d00692

    ScanJob({BAA53BE2-9FE1-48B6-89A7-0D60BC07ED50}): CScanJobManager::OnScanComplete- failed at CScanJob::OnScanComplete with error=0x87d00692


    Conclusion:
    We have a Group policy in place applying a specific update service location; this location is applied in several policies over several OU's. The single policy that was resulting in  0x87d00692 has had the update service location removed and handed management over to the SCCM client. The error has been been removed from the log and the client updated almost instantly.

    "Specify intranet Microsoft update service location"

    Thursday, August 3, 2017

    SCCM SUP WSUS Pool keeps stopping or the server is unresponsive

    SCCM SUP WSUS Pool keeps stopping or the server is unresponsive

    Scenario: Our WSUS/SUP had become unresponsive and the decision to reinstall the server role was made. After the Site server had been reinstalled  I became aware that Windows Defender updates were failing to update (3 days old) and even though the updates were sync'd, downloaded, and deployed in SCCM the client was still unable to receive them.


    Client Log analysis:

    ScanAgent.log

    ScanJob({999C9FFA-A463-4BE8-8771-67EE96D4140B}): CScanJob::OnScanComplete -Scan Failed with Error=0x80240440
    ScanJob({999C9FFA-A463-4BE8-8771-67EE96D4140B}): CScanJobManager::OnScanComplete- failed at CScanJob::OnScanComplete with error=0x80240440



    Update Deployment.log

    Job error (0x80240440) received for assignment ({bf7a48e6-d220-4070-bb9b-ecc239107584}) action        UpdatesDeploymentAgent       
    Updates will not be made available       

    WUAHandler.log

    Async searching of updates using WUAgent started.       
    Async searching completed.       
    OnSearchComplete - Failed to end search job. Error = 0x8024401c.        
    Scan failed with error = 0x8024401c.        
    Its a WSUS Update Source type ({3AAB6A76-CE2D-4E8A-9F11-741AE69677A2}), adding it.        
    OS Version is 6.3.9600   
    Existing WUA Managed server was already set (http://CMSUP.domain.co.uk:8530), skipping Group Policy registration.        

    Added Update Source ({3AAB6A76-CE2D-4E8A-9F11-741AE69677A2}) of content type: 2        

    WindowsUpdate.log

    Report WARNING: CSerializationHelper:: InitSerialize failed : 0x80070002
    AU        WARNING: Failed to get Wu Exemption info from NLM, assuming not exempt, error = 0x80070032
    WS        WARNING: Nws Failure: errorCode=0x803d0006
    WS        WARNING: Original error code: 0x80072ee2
    WS        WARNING: There was an error communicating with the endpoint at 'http://CMSUP.domain.co.uk:8530/ClientWebService/client.asmx'.
    WS        WARNING: There was an error receiving the HTTP reply.
    WS        WARNING: The operation did not complete within the time allotted.
    WS        WARNING: The operation timed out


    Analysis Results:
    Since I removed the role and put it back it was like the role had been installed for the first time.  Every client within SCCM (9k+) would need to do a Full Software Update & scan cycle. This would generate a heavy load on the ISS pool on the SUP server.  It is very important to configure the pool correctly otherwise the server will stop responding to clients as the client will receive "Service Unavailable" responses similar to DOS.

    WSUS Pool Config
    Queue Length: 25000
    Limit (Percent) 70
    Limit Action Throttle
    Maximum workers (0 Numa) (1 Default) (2 if you have the server resource, not NUMA)
    "Service Unavailable" Response TcpLevel
    Failure Interval (Minutes) 30
    Maximum Failures 60
    Private Memory Limit (KB) 0


    Conclusion
     Once we had the SUP role reinstalled synchronizing with SCCM (see WCM.log, Wsyncmgr.log) as well as the correct WSUS pool settings, we notice the server was still spiking CPU with ISS workers taking the full 70% limit without breaks.
    Upon reviewing the Client Policy (within SCCM Client Settings) for the estate the "Software Update Scan schedule" was set to 1 day (not the recommended 7 days). This had the affect of overloading the WSUS server with 503 errors in the IIS log "service unavailable". As the schedule was every day the server could not get through the backlog before the whole process began again.
    After setting the scan schedule to 7 days, and as clients were checking for policy every hour we notice a steady decline in CPU activity within the hour and all clients were able to complete there scan Software Update & scan cycle  and all clients were able to download Software updates.