I have been working on this issue for some time and it was always failing with different kind of errors, i consulted Microsoft Team as well and finally it worked with me. I am using DPM 2012 but it shouldn't differ with any earlier DPM version as 2010. The main blocking issue here is the TMG component on the UAG server. Remember that its highly recommended not to touch the TMG configuration on the UAG however this is one of the rare cases that need the administrator to tweak some TMG settings. To enable the DPM to backup UAG and install the client you need to do the following (Make sure to take Full backup from your TMG settings and Rules):
1. Ensure the File and Printer sharing is checked/Enabled on the UAG internal Network card.
2. From the TMG console- Firewall Policy. On the right pane click show system Policy rule
3. You need to disable system Rule number 2 (Allow Remote Management from selected computers using MMC) by Right clicking the rule and edit system policy, I am assuming the default TMG rules are not touched before.
4. You need to disable System Rule number 22 (Allow RPC from Forefront TMG to Trusted servers)
5. From the Right Pane, in the toolbox section create a new Protocol under user defined. The Protocol parameters as follows:
Primary connection: Type: TCP, Direction: Outbound, Port range: 135-135
Secondary Connection: Type: TCP, Direction: Outbound, Port range: 1024 - 65535
6. Final Step, create a new Access Rule (Make sure to move it to the top). Allow - All outbound traffic except selected (Choose - RPC All interfaces) - From DPM server (Create computer object with DPM IP address)- To Local Host (UAG server) - All users................etc
Save the Settings and ensure they are Synced from the monitoring tab. Now try to install the Agent from the DPM on the UAG server and take a simple test backup.
1. Ensure the File and Printer sharing is checked/Enabled on the UAG internal Network card.
2. From the TMG console- Firewall Policy. On the right pane click show system Policy rule
3. You need to disable system Rule number 2 (Allow Remote Management from selected computers using MMC) by Right clicking the rule and edit system policy, I am assuming the default TMG rules are not touched before.
4. You need to disable System Rule number 22 (Allow RPC from Forefront TMG to Trusted servers)
5. From the Right Pane, in the toolbox section create a new Protocol under user defined. The Protocol parameters as follows:
Primary connection: Type: TCP, Direction: Outbound, Port range: 135-135
Secondary Connection: Type: TCP, Direction: Outbound, Port range: 1024 - 65535
6. Final Step, create a new Access Rule (Make sure to move it to the top). Allow - All outbound traffic except selected (Choose - RPC All interfaces) - From DPM server (Create computer object with DPM IP address)- To Local Host (UAG server) - All users................etc
Save the Settings and ensure they are Synced from the monitoring tab. Now try to install the Agent from the DPM on the UAG server and take a simple test backup.
