Date: Wed, 25 Sep 91 22:52:02 CDT
From: varney@ihlpf.att.com (Al L Varney)
Subject: Re: How Do They Change Equipment Without Disrupting Service?
Organization: AT&T Network Systems

In article <telecom11.765.4@eecs.nwu.edu> John Higdon <john@zygot.
ati.com> writes:

> On the recent cut from #5 crossbar to #5ESS in the ANdrews (San Jose)
> office, anyone on the telephone at 11:33 pm was unceremoniously
> dropped. The line was dead for about two minutes, then dial tone from
> the new switch appeared. IntraLATA calls out were completed promptly
> and properly, but incoming calls from outside of the physical central
> office did not complete, and calls from the old co-located 1ESS were
> completed properly from the moment of cutover.  Calling in from other
> offices generally resulted in silence. Outgoing calls requiring an IEC
> could not be made.

> As the clock approached midnight (the scheduled cut time), incoming
> intra-LATA calls became possible from more and more offices ...

A quick tutorial on Switch cut-overs ... note that all differ from the
norm in various ways!

The goal of Switch cut-overs is to first change the LINES over to the
new Switch, then the TRUNKS, then modify any routing at other COs to
use any new TRUNKS.  For ESS(tm) COs, it is possible to also preserve
LINE-to-LINE calls, and outside calls.

For pre-ESS old COs, a new ESS CO will have connections bridged
("Y"-connected) to all LINES at the Main Distributing Frame (MDF)
weeks in advance of cut-over.  These LINES are marked as
"inactive-precut", and will NOT normally be supplied battery/ground or
dial-tone.  A test program in the new CO will run through designated
ranges of telephone numbers, selecting IDLE (on-hook) "inactive-precut" 
lines for testing.  For each selected line, a Test connection is made
to the line via the old CO Test Desk (using a special test trunk from
the new CO to the old CO Test Desk).  Using the Test Desk connection,
battery reversal and other changes are made to the line; any such
changes should be seen at the new CO line appearance.  In effect, the
test is to assure that the two "arms" of the "Y" connection are
physically connected to the same pair of wires at the MDF.  Note that
any "off-hook" from the customer side of the MDF will, after a second
or so, result in test termination and dial tone from the old CO.  The
software will usually remember lines that couldn't be tested (or failed) 
and will periodically retest them.

A similar mechanism is used for TRUNKS, except it only affects
incoming traffic (outgoing calls in a pre-cut office MUST BE test
calls, right?).  TRUNKS marked "precut" will be forced into a "bypass"
state, appearing as an open circuit to the old CO and the far-end CO.
Testing is manually initiated, and requires the circuit to be busied
and forced to bypass or disconnected at the old CO.  Testing from the
new CO and the far-end CO can occur.  Regardless of test results, the
"precut" trunk will automatically be forced back to "bypass" after a
period of non-testing.

If the old CO is non-digital, it is common for trunks to not be
cross-connected at the analog level, but to either use a patch panel
to bridge T1 facilities directly into the new CO at cut or to add new
facilities just for the new CO.  For the San Jose cut (speculation; I
wasn't involved), it seems likely some new T1 facilities were run.

At the time of cut-over from a non-ESS CO, the LINE cut-over involves
typing a message at the new CO (e.g., SET:STATE=POSTCUT) just after
cutting the MDF-to-old-CO wires.  There is no means other than
bolt-cutters, etc. to stop dial tone from the old CO!  Now the TRUNKS
at the old CO are also cut or patched to the new CO. If new facilities 
were used, the far-end offices must all be changed to use the new
trunks for calls incoming to the new CO.  Outgoing calls can occur
immediately, even before the far-end COs have been updated.
In-progress calls at cut are obviously lost.  There is obviously no
going back to the old CO after the bolt-cutters do their job.

For old ESS CO replacements, similar methods are used.  The major
differences are that LINES in the old CO can be marked "inactive-
postcut".  When the CUT message is input to the new CO, it will toggle
a private lead to the old CO, automatically forcing it to also become
"postcut".  However, postcut in the old CO means it should stop
offering dial tone to new off-hook LINES.  It does NOT drop existing
connections; they can be allowed to die off over the next few hours.
If problems develop, both offices can be changed back to "precut" with
a single input message at the new CO.

The other major capability offered by old ESS COs for cut-over support
is the ability to build a "temporary" tandem trunking arrangement from
the old CO to the new CO.  Incoming calls to either CO destined for
"inactive" lines will automatically be tandemed to the other CO.  This
means that, before cut-over time, some incoming calls actually arrive
at the new CO and tandem to the old CO for completion.  After
cut-over, incoming calls to the old CO tandem over to the new CO.  The
decision to tandem is made only by examining the "inactive" status of
a line and the current office state (precut/postcut) just before
connecting to the line.  NO active call is affected by the pre-to-
post-cut office state change, only new originations and terminations.
So it is possible (and likely) that a CO replacement can occur without
ANY customer being able to detect the moment of "cut"!!  There are
certainly few other computer replacements that can be this transparent
to active users!!!!

Note that the issue of testing and cutover of ISDN lines is not as
easily solved, since no equivalent to a "Y" connection is obvious for
the D-channel.  They must currently be tested in a manner similar to
Digital TRUNKS, and drop calls at the moment of cut-over of the ISDN
facilities.  Of course, there aren't a lot of ISDN COs being replaced,
yet ....

Hope this article passes the NEW Telecom test.


Al Varney, AT&T Network Systems, Lisle, IL

Date: Mon,  4 Nov 91 13:37:06 CST
From: varney@ihlpf.att.com (Al L Varney)
Subject: Re: Oldest 1ESS in USA Retires
Organization: AT&T Network Systems

In article <telecom11.879.6@eecs.nwu.edu> jj1028@homxc.att.com
(Maurice R Baker) writes:

> From {Bell Labs News} of 10/28/91:
>	----Picture of a row of craftpersons with bolt cutters severing
>		many multi-pair cables---

>      The cutover was accomplished in three steps.  First, the new
> 5ESS was installed and hooked up to the incoming lines -- which were
> also still going to the old switch -- but not powered up.  Then, at
> 1:00 AM, when there is relatively little traffic on the old switch,
> the cables coming in to the 1ESS were quickly cut.  Once all the lines
> were cut, the new switch was turned on.  The whole operation took only
> a few minutes, and emergency services agencies were notified ....

   I've written before about the ability of ESS(tm) switches to
cut-over lines (one or more NXXs, or an entire office) without the
need to drop any calls or cut cables.  Each cut-over is handled on a
case-by-case basis, so some cut-overs do not use all the capabilities
available.

   In the Succasunna cut-over, the sequence involved was:

1) Lines were marked for cut-over in each switch, as usual, except NJB
marked the old switch lines to be "in-active@pre-cut" and set the
switch status as "post-cut".  The line cut-over normally sets the both
switches to "pre-cut".  The major difference in procedure involves the
treatment of "unassigned" lines in the old switch.

2) All trunks were previously set up to connect via T1 patch panels to
the new switch.  Patch cords were used to connect to the old switch
facilities.  

3) At about 1 A.M., set the old office status to "pre-cut", making the
switch not recognize new "off-hooks".  Set the new office to "pre-cut",
making new off-hook requests result in dial tone from the new switch.

4) Pull the patch cords on the T1 facilities.  Drops any inprogess
trunk calls in the old switch.

5) A while later, cut the line-side cables from the old switch to the
MDF.  Drops any old line-to-line calls in the old switch.

6) Some days later, when service is completely verified, power off and
remove the old switch.

Step 5 is not usually needed until the new switch operation is
verified; NJB felt the age of the equipment, particularly in the
contacts that are used to remove the "off-hook" detection at this No.
1 ESS, might result in some added induction loading on lines after
cut-over.  Once calls appeared to be working on the new switch, the
cables were cut.

And in article <telecom11.884.5@eecs.nwu.edu> cml@cs.UMD.EDU,
(Christopher Lott) writes:

> If this is obvious, please forgive me.  I don't understand why the old
> 1ESS switch wasn't just powered down about the same time that the new
> one was powered up.  Wouldn't this do the same thing??  Seems awfully
> final to cut the cables.  What if the new switch doesn't work right?
> I must be missing something.

  Briefly: (remember Succasunna was large, say gymnasium-sized).
Powering off a switch will not reliably remove ground or forms of
resistance/inductance from lines, since the non-power state of relays
and cross-points is not predictable.

  The old switch would act as a load on the lines to the new switch.

  The alarms set off might mask real alarms from the new switch or
other systems, and also require a lot of paperwork later.

  Power spikes induced by any massive load removal is not a Good Thing
to do to power sources, especially since the sources probably also
power the new switch.

  The No. 1 ESS switches were not built to be turned off/on
frequently.  If the new switch doesn't work, a quick massive power-up
of all the redundant power sources might damage lots of circuits.
Normally, power-up takes lots of time, as in hours!  This could be a
lot more final than cutting cables, which, as I mentioned, was done
only after new switch calls were tested.


Al Varney, AT&T Network Systems, Lisle, IL

Date: Mon, 11 Nov 91 11:56:15 CST
From: varney@ihlpf.att.com (Alan L Varney)
Subject: Re: Oldest 1ESS in USA Retires
Organization: AT&T Network Systems

In article <telecom11.910.6@eecs.nwu.edu> cornutt@freedom.msfc.
nasa.gov (David Cornutt) writes:

> Thanks for an interesting article, Al.  Now let me ask you a question
> about cutover.  I'm looking at a flyer from Tekelec ...  they call
> the Cutover Device.  From the description of it, and my limited
> knowledge, I take it that it's intended for cutting over toll
> switches.

> It appears to be a device with which two switches can be connected to
> an SS7 net so that they appear to be one device.  On command,
> apparently it starts routing call setup requests to the new switch,
> while still handling messages concerning existing traffic to/from the
> switch which is being cut over.  I assume that it keeps track of
> remaining calls on the old switch and notifies the operator when all
> traffic has been cut over.  Is this an accurate summary?  Am I correct
> in the assumption that it is only for toll switches, or can it be used
> for COs too?  (And if so, how are the individual pairs cut over?  Come
> to think of it, when all the analog systems are gone, presumably there
> won't be any pairs coming directly to the switch, just trunks.  At
> that point, will you even need a device like this any more?)

   From my limited understanding of Tekelec's SS7 device, it serves as
a router of incoming Trunk Signaling (actually ISUP) SS7 messages,
while perhaps intercepting inappropriate outgoing messages.  It plays
no role in the physical cut-over of trunks, and is not involved in
lines at all.

   To understand the need for this device, one has to understand the
routing of ISUP SS7 messages to individual switches.  The SS7 packet
protocol essentially identifies a trunk/circuit with a far-end Point
Code (PC, 24 bits) and a Circuit Identification Code (CIC, 14 bits).
Since the SS7 protocol was not designed with network re-arrangements
in mind, only one node (switch) can be associated with a Point Code.

   So now the problem with cut-over is that either the new switch has
to assume the PC of the old switch (Tekelec's box supports this) or
one has to "flash cut" some trunks at every connecting switch to the
different PC of the new switch.  The first method is much easier at
cut-over time. Before cut-over, one would like to move trunks in small
groups to the new switch, test them, then move them back.  With SS7,
this is difficult if the switch is "assuming" the original Point Code
because there is no easy way to connect the new switch to the SS7
network (with the new Point Code) and still allow most traffic through
the old switch.

   The box (I'm guessing here) must connect into the signalling links
coming into both the old and new switches, allowing both to have the
same PC.  All outgoing traffic from the new switch would be blocked.
Then a small number of PC/CIC identifiers are marked "under test", the
corresponding trunks are blocked from traffic, allowing the trunks to
be moved and tested.  In this mode, the box would "re-route" ISUP
messages for specific trunks to/from the new switch.  At cut-over, I
don't see the box providing any traffic re-routing; it probably just
sends everything to the new switch.  (After all, at cut-over, all the
trunks are typically reconnected to the new switch.  There are few
cases anymore where duplicate trunks are provisioned, allowing calls
to terminate normally.)

   A similar mechanism is available from AT&T when the old switch is a
1A ESS(tm) switch and the new one is a 5ESS(reg) switch.  No special
hardware is involved, and coordination of test trunks is not as
critical.  There is even some interest in the Standards groups for a
non-proprietary solution, but don't look for anything soon ...


Al Varney, AT&T Network Systems, Lisle, IL; Above info. is my own opinion.

Date: Sun, 14 Jun 92 13:35:07 CDT
From: varney@ihlpf.att.com (Alan L Varney)
Subject: Re: Six Points - Response
Organization: AT&T Network Systems, Lisle, IL

TELECOM Moderator notes, in response to John Higdon:

> [Moderator's Note: Actually John, Centrex goes back further than
> that. ...  Another early Centrex user here was the {Chicago
> Tribune} (312-222) dating from the middle sixties, also on xbar.

   I think this was the LAST xbar Centrex in IBT.  I know we sat with
lots of worried folks the night (in 1977 or '78) we cut it to a 1A
ESS (TM) switch.  Seems there were lots of "sleeve-lead" and other
un-documented wires connected to the {Tribune}, and IBT was prepared
for lots of complaints from the "Tower" after the cut-over.  As I
recall, there were NO complaints; the Network Maintenance folks
figured that the staff at the {Tribune} had changed enough that they
had forgotten how to use those "special hand-wired" features!


Al Varney - just MY opinion.